2026-02-22T22:19:10.405 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T22:19:10.422 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T22:19:10.498 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64214 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '64214' kernel: branch: distro kdb: true sha1: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5907 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJUUMDAl+fxwxFAfCUBaNIv4q2tfKtYsmd2PzO1UF7dWLNKMulq50OdY/UNewbSiUqmy6q6MzhWsGfGAvpQomq0= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:40 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T22:19:10.498 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T22:19:10.499 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T22:19:10.499 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T22:19:10.500 INFO:teuthology.task.internal:Checking packages... 2026-02-22T22:19:10.500 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T22:19:10.500 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T22:19:10.500 INFO:teuthology.packaging:ref: None 2026-02-22T22:19:10.500 INFO:teuthology.packaging:tag: None 2026-02-22T22:19:10.500 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:19:10.500 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:19:10.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:19:10.622 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689.gf90f4eae 2026-02-22T22:19:10.623 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T22:19:10.641 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T22:19:10.641 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T22:19:10.654 INFO:teuthology.task.internal:Saving configuration 2026-02-22T22:19:10.660 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T22:19:10.672 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T22:19:10.752 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial123.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64214', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 22:15:45.298190', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJUUMDAl+fxwxFAfCUBaNIv4q2tfKtYsmd2PzO1UF7dWLNKMulq50OdY/UNewbSiUqmy6q6MzhWsGfGAvpQomq0='} 2026-02-22T22:19:10.752 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T22:19:10.764 INFO:teuthology.task.internal:roles: ubuntu@trial123.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-22T22:19:10.764 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T22:19:10.846 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x739b809f2160>, signals=[15]) 2026-02-22T22:19:10.846 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T22:19:10.858 INFO:teuthology.task.internal:Opening connections... 2026-02-22T22:19:10.858 DEBUG:teuthology.task.internal:connecting to ubuntu@trial123.front.sepia.ceph.com 2026-02-22T22:19:10.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T22:19:10.923 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T22:19:10.935 DEBUG:teuthology.orchestra.run.trial123:> uname -m 2026-02-22T22:19:11.004 INFO:teuthology.orchestra.run.trial123.stdout:x86_64 2026-02-22T22:19:11.004 DEBUG:teuthology.orchestra.run.trial123:> cat /etc/os-release 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:NAME="CentOS Stream" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:VERSION="9" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:ID="centos" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:ID_LIKE="rhel fedora" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:VERSION_ID="9" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:PLATFORM_ID="platform:el9" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:ANSI_COLOR="0;31" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:LOGO="fedora-logo-icon" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:HOME_URL="https://centos.org/" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-22T22:19:11.059 INFO:teuthology.orchestra.run.trial123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-22T22:19:11.060 INFO:teuthology.lock.ops:Updating trial123.front.sepia.ceph.com on lock server 2026-02-22T22:19:11.144 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T22:19:11.158 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T22:19:11.170 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T22:19:11.170 DEBUG:teuthology.orchestra.run.trial123:> test '!' -e /home/ubuntu/cephtest 2026-02-22T22:19:11.186 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T22:19:11.197 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T22:19:11.197 DEBUG:teuthology.orchestra.run.trial123:> test -z $(ls -A /var/lib/ceph) 2026-02-22T22:19:11.242 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T22:19:11.309 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T22:19:11.328 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T22:19:11.328 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T22:19:11.328 DEBUG:teuthology.orchestra.run.trial123:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T22:19:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:19:11.343 DEBUG:teuthology.orchestra.run.trial123:> uname -r 2026-02-22T22:19:11.399 INFO:teuthology.orchestra.run.trial123.stdout:5.14.0-681.el9.x86_64 2026-02-22T22:19:11.399 INFO:teuthology.task.kernel:Running kernel on trial123: 5.14.0-681.el9.x86_64 2026-02-22T22:19:11.399 DEBUG:teuthology.orchestra.run.trial123:> sudo yum install -y kernel 2026-02-22T22:19:14.150 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.9 MB 00:02 2026-02-22T22:19:16.741 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-22T22:19:20.089 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 7.8 MB 00:00 2026-02-22T22:19:21.927 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-22T22:19:23.205 INFO:teuthology.orchestra.run.trial123.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-22T22:19:26.448 INFO:teuthology.orchestra.run.trial123.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-22T22:19:27.471 INFO:teuthology.orchestra.run.trial123.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-22T22:19:27.472 INFO:teuthology.orchestra.run.trial123.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-22T22:19:27.490 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:19:27.491 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:19:27.491 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:19:27.516 DEBUG:teuthology.orchestra.run.trial123:> echo no | sudo yum reinstall kernel || true 2026-02-22T22:19:27.814 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:01 ago on Sun 22 Feb 2026 10:19:26 PM UTC. 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout: Package Architecture Version Repository Size 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout:Reinstalling: 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:19:27.895 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:19:27.896 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:19:27.896 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 803 k 2026-02-22T22:19:27.896 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 0 2026-02-22T22:19:27.896 INFO:teuthology.orchestra.run.trial123.stderr:Operation aborted. 2026-02-22T22:19:27.915 INFO:teuthology.orchestra.run.trial123.stdout:Is this ok [y/N]: 2026-02-22T22:19:27.915 DEBUG:teuthology.orchestra.run.trial123:> sudo yum reinstall -y kernel || true 2026-02-22T22:19:28.220 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:02 ago on Sun 22 Feb 2026 10:19:26 PM UTC. 2026-02-22T22:19:28.304 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:19:28.304 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:19:28.304 INFO:teuthology.orchestra.run.trial123.stdout: Package Architecture Version Repository Size 2026-02-22T22:19:28.304 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:19:28.304 INFO:teuthology.orchestra.run.trial123.stdout:Reinstalling: 2026-02-22T22:19:28.304 INFO:teuthology.orchestra.run.trial123.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-22T22:19:28.304 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:19:28.305 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:19:28.305 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:19:28.305 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:19:28.305 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 803 k 2026-02-22T22:19:28.305 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 0 2026-02-22T22:19:28.305 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:19:28.588 INFO:teuthology.orchestra.run.trial123.stdout:kernel-5.14.0-681.el9.x86_64.rpm 30 MB/s | 803 kB 00:00 2026-02-22T22:19:28.588 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:19:28.588 INFO:teuthology.orchestra.run.trial123.stdout:Total 2.8 MB/s | 803 kB 00:00 2026-02-22T22:19:28.598 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:19:28.632 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:19:28.632 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:19:28.637 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:19:28.637 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:19:28.662 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:19:28.665 INFO:teuthology.orchestra.run.trial123.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T22:19:28.666 INFO:teuthology.orchestra.run.trial123.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T22:19:28.773 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T22:19:28.774 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-22T22:19:28.830 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-22T22:19:28.830 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:19:28.830 INFO:teuthology.orchestra.run.trial123.stdout:Reinstalled: 2026-02-22T22:19:28.830 INFO:teuthology.orchestra.run.trial123.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-22T22:19:28.831 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:19:28.831 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:19:28.858 DEBUG:teuthology.orchestra.run.trial123:> rpm -q kernel | sort -rV | head -n 1 2026-02-22T22:19:28.881 INFO:teuthology.orchestra.run.trial123.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-22T22:19:28.882 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-22T22:19:28.882 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T22:19:28.883 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-22T22:19:28.883 DEBUG:teuthology.orchestra.run.trial123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T22:19:28.947 INFO:teuthology.orchestra.run.trial123.stdout:ttyS1 2026-02-22T22:19:28.974 DEBUG:teuthology.parallel:result is None 2026-02-22T22:19:28.974 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T22:19:28.986 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T22:19:28.986 DEBUG:teuthology.orchestra.run.trial123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T22:19:29.030 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T22:19:29.042 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T22:19:29.055 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T22:19:29.055 DEBUG:teuthology.orchestra.run.trial123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T22:19:29.090 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T22:19:29.102 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T22:19:29.103 DEBUG:teuthology.orchestra.run.trial123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T22:19:29.150 INFO:teuthology.orchestra.run.trial123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:19:29.158 INFO:teuthology.orchestra.run.trial123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:19:29.160 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T22:19:29.171 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T22:19:29.171 DEBUG:teuthology.orchestra.run.trial123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T22:19:29.223 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T22:19:29.239 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T22:19:29.240 DEBUG:teuthology.orchestra.run.trial123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T22:19:29.280 DEBUG:teuthology.orchestra.run.trial123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T22:19:29.441 DEBUG:teuthology.orchestra.run.trial123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T22:19:29.464 DEBUG:teuthology.orchestra.run.trial123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T22:19:29.613 DEBUG:teuthology.orchestra.run.trial123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T22:19:29.635 DEBUG:teuthology.orchestra.run.trial123:> set -ex 2026-02-22T22:19:29.635 DEBUG:teuthology.orchestra.run.trial123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T22:19:29.698 DEBUG:teuthology.orchestra.run.trial123:> sudo service rsyslog restart 2026-02-22T22:19:29.764 INFO:teuthology.orchestra.run.trial123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T22:19:30.043 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T22:19:30.055 INFO:teuthology.task.internal:Starting timer... 2026-02-22T22:19:30.055 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T22:19:30.077 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T22:19:30.098 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2026-02-22T22:19:30.098 DEBUG:teuthology.orchestra.run.trial123:> sudo service auditd rotate 2026-02-22T22:19:30.126 INFO:teuthology.orchestra.run.trial123.stdout:Rotating logs: 2026-02-22T22:19:30.128 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T22:19:30.129 DEBUG:teuthology.orchestra.run.trial123:> /usr/sbin/getenforce 2026-02-22T22:19:30.185 INFO:teuthology.orchestra.run.trial123.stdout:Permissive 2026-02-22T22:19:30.185 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial123.front.sepia.ceph.com': 'permissive'} 2026-02-22T22:19:30.185 DEBUG:teuthology.orchestra.run.trial123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T22:19:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:19:30.250 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T22:19:30.250 DEBUG:teuthology.orchestra.run.trial123:> sudo /usr/sbin/setenforce permissive 2026-02-22T22:19:30.313 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T22:19:30.329 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T22:19:30.341 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T22:19:30.342 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T22:22:09.753 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial123.front.sepia.ceph.com')] 2026-02-22T22:22:09.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial123.front.sepia.ceph.com' 2026-02-22T22:22:09.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T22:22:09.876 DEBUG:teuthology.orchestra.run.trial123:> true 2026-02-22T22:22:09.956 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial123.front.sepia.ceph.com' 2026-02-22T22:22:09.957 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T22:22:09.971 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T22:22:09.971 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T22:22:09.971 DEBUG:teuthology.orchestra.run.trial123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T22:22:10.023 INFO:teuthology.orchestra.run.trial123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-22T22:22:10.035 INFO:teuthology.orchestra.run.trial123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-22T22:22:10.056 INFO:teuthology.orchestra.run.trial123.stderr:sudo: ntpd: command not found 2026-02-22T22:22:10.066 INFO:teuthology.orchestra.run.trial123.stdout:506 Cannot talk to daemon 2026-02-22T22:22:10.077 INFO:teuthology.orchestra.run.trial123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-22T22:22:10.090 INFO:teuthology.orchestra.run.trial123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-22T22:22:10.137 INFO:teuthology.orchestra.run.trial123.stderr:bash: line 1: ntpq: command not found 2026-02-22T22:22:10.155 INFO:teuthology.orchestra.run.trial123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T22:22:10.155 INFO:teuthology.orchestra.run.trial123.stdout:=============================================================================== 2026-02-22T22:22:10.155 INFO:teuthology.orchestra.run.trial123.stdout:^? LAX.CALTICK.NET 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T22:22:10.155 INFO:teuthology.orchestra.run.trial123.stdout:^? 50.205.57.38 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T22:22:10.155 INFO:teuthology.orchestra.run.trial123.stdout:^? clock02.cymru.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T22:22:10.155 INFO:teuthology.orchestra.run.trial123.stdout:^? c-73-238-61-241.hsd1.nh.> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-22T22:22:10.156 INFO:teuthology.run_tasks:Running task install... 2026-02-22T22:22:10.169 DEBUG:teuthology.task.install:project ceph 2026-02-22T22:22:10.169 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T22:22:10.170 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T22:22:10.170 INFO:teuthology.task.install:Using flavor: default 2026-02-22T22:22:10.179 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T22:22:10.179 INFO:teuthology.task.install:extra packages: [] 2026-02-22T22:22:10.179 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579', 'tag': None, 'wait_for_package': False} 2026-02-22T22:22:10.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:22:10.340 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/ 2026-02-22T22:22:10.341 INFO:teuthology.task.install.rpm:Package version is 19.2.3-689.gf90f4eae 2026-02-22T22:22:10.483 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-22T22:22:10.483 DEBUG:teuthology.orchestra.run.trial123:> set -ex 2026-02-22T22:22:10.483 DEBUG:teuthology.orchestra.run.trial123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-22T22:22:10.509 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-22T22:22:10.509 DEBUG:teuthology.orchestra.run.trial123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f90f4eaebb28b619cf76e80cbbe997656da69579/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-22T22:22:10.574 DEBUG:teuthology.orchestra.run.trial123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-22T22:22:10.649 DEBUG:teuthology.orchestra.run.trial123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-22T22:22:10.713 INFO:teuthology.orchestra.run.trial123.stdout:check_obsoletes = 1 2026-02-22T22:22:10.715 DEBUG:teuthology.orchestra.run.trial123:> sudo yum clean all 2026-02-22T22:22:10.872 INFO:teuthology.orchestra.run.trial123.stdout:41 files removed 2026-02-22T22:22:10.886 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-radosgw 2026-02-22T22:22:11.156 INFO:teuthology.orchestra.run.trial123.stdout:ceph packages for x86_64 667 kB/s | 86 kB 00:00 2026-02-22T22:22:11.287 INFO:teuthology.orchestra.run.trial123.stdout:ceph noarch packages 113 kB/s | 12 kB 00:00 2026-02-22T22:22:11.416 INFO:teuthology.orchestra.run.trial123.stdout:ceph source packages 18 kB/s | 1.9 kB 00:00 2026-02-22T22:22:12.731 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.9 MB 00:01 2026-02-22T22:22:15.907 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-22T22:22:18.538 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - CRB 30 MB/s | 7.8 MB 00:00 2026-02-22T22:22:20.367 INFO:teuthology.orchestra.run.trial123.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 20 kB 00:01 2026-02-22T22:22:21.230 INFO:teuthology.orchestra.run.trial123.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2026-02-22T22:22:24.363 INFO:teuthology.orchestra.run.trial123.stdout:lab-extras 1.9 MB/s | 50 kB 00:00 2026-02-22T22:22:25.557 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:22:25.558 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:22:25.558 INFO:teuthology.orchestra.run.trial123.stdout: Package Arch Version Repository Size 2026-02-22T22:22:25.558 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:22:25.558 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:22:25.558 INFO:teuthology.orchestra.run.trial123.stdout: ceph-radosgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 11 M 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout:Upgrading: 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: librados2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.4 M 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: librbd1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.2 M 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout:Installing dependencies: 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: ceph-base x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.5 M 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 22 M 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: ceph-selinux x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 25 k 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-22T22:22:25.559 INFO:teuthology.orchestra.run.trial123.stdout: libcephfs2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 999 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: libradosstriper1 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 503 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: librgw2 x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 5.4 M 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-22T22:22:25.560 INFO:teuthology.orchestra.run.trial123.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: python3-ceph-argparse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 45 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: python3-ceph-common x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 142 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: python3-cephfs x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 165 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: python3-rados x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 322 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: python3-rbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 303 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: python3-rgw x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 100 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout:Installing weak dependencies: 2026-02-22T22:22:25.561 INFO:teuthology.orchestra.run.trial123.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-22T22:22:25.562 INFO:teuthology.orchestra.run.trial123.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-22T22:22:25.562 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:25.562 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:22:25.562 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:22:25.562 INFO:teuthology.orchestra.run.trial123.stdout:Install 33 Packages 2026-02-22T22:22:25.562 INFO:teuthology.orchestra.run.trial123.stdout:Upgrade 2 Packages 2026-02-22T22:22:25.562 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:25.563 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 63 M 2026-02-22T22:22:25.563 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:22:26.226 INFO:teuthology.orchestra.run.trial123.stdout:(1/35): ceph-radosgw-19.2.3-689.gf90f4eae.el9.x 25 MB/s | 11 MB 00:00 2026-02-22T22:22:26.232 INFO:teuthology.orchestra.run.trial123.stdout:(2/35): ceph-base-19.2.3-689.gf90f4eae.el9.x86_ 12 MB/s | 5.5 MB 00:00 2026-02-22T22:22:26.237 INFO:teuthology.orchestra.run.trial123.stdout:(3/35): ceph-selinux-19.2.3-689.gf90f4eae.el9.x 2.1 MB/s | 25 kB 00:00 2026-02-22T22:22:26.267 INFO:teuthology.orchestra.run.trial123.stdout:(4/35): libradosstriper1-19.2.3-689.gf90f4eae.e 17 MB/s | 503 kB 00:00 2026-02-22T22:22:26.284 INFO:teuthology.orchestra.run.trial123.stdout:(5/35): libcephfs2-19.2.3-689.gf90f4eae.el9.x86 19 MB/s | 999 kB 00:00 2026-02-22T22:22:26.320 INFO:teuthology.orchestra.run.trial123.stdout:(6/35): python3-ceph-argparse-19.2.3-689.gf90f4 1.2 MB/s | 45 kB 00:00 2026-02-22T22:22:26.377 INFO:teuthology.orchestra.run.trial123.stdout:(7/35): python3-ceph-common-19.2.3-689.gf90f4ea 2.5 MB/s | 142 kB 00:00 2026-02-22T22:22:26.408 INFO:teuthology.orchestra.run.trial123.stdout:(8/35): librgw2-19.2.3-689.gf90f4eae.el9.x86_64 38 MB/s | 5.4 MB 00:00 2026-02-22T22:22:26.427 INFO:teuthology.orchestra.run.trial123.stdout:(9/35): python3-cephfs-19.2.3-689.gf90f4eae.el9 3.2 MB/s | 165 kB 00:00 2026-02-22T22:22:26.431 INFO:teuthology.orchestra.run.trial123.stdout:(10/35): python3-rados-19.2.3-689.gf90f4eae.el9 14 MB/s | 322 kB 00:00 2026-02-22T22:22:26.444 INFO:teuthology.orchestra.run.trial123.stdout:(11/35): python3-rgw-19.2.3-689.gf90f4eae.el9.x 7.6 MB/s | 100 kB 00:00 2026-02-22T22:22:26.465 INFO:teuthology.orchestra.run.trial123.stdout:(12/35): python3-rbd-19.2.3-689.gf90f4eae.el9.x 7.8 MB/s | 303 kB 00:00 2026-02-22T22:22:26.503 INFO:teuthology.orchestra.run.trial123.stdout:(13/35): mailcap-2.1.49-5.el9.noarch.rpm 563 kB/s | 33 kB 00:00 2026-02-22T22:22:26.531 INFO:teuthology.orchestra.run.trial123.stdout:(14/35): pciutils-3.7.0-7.el9.x86_64.rpm 3.2 MB/s | 93 kB 00:00 2026-02-22T22:22:26.565 INFO:teuthology.orchestra.run.trial123.stdout:(15/35): nvme-cli-2.16-1.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2026-02-22T22:22:26.776 INFO:teuthology.orchestra.run.trial123.stdout:(16/35): ceph-common-19.2.3-689.gf90f4eae.el9.x 22 MB/s | 22 MB 00:00 2026-02-22T22:22:27.256 INFO:teuthology.orchestra.run.trial123.stdout:(17/35): boost-program-options-1.75.0-13.el9.x8 144 kB/s | 104 kB 00:00 2026-02-22T22:22:27.302 INFO:teuthology.orchestra.run.trial123.stdout:(18/35): libnbd-1.20.3-4.el9.x86_64.rpm 222 kB/s | 164 kB 00:00 2026-02-22T22:22:27.500 INFO:teuthology.orchestra.run.trial123.stdout:(19/35): libpmemobj-1.12.1-1.el9.x86_64.rpm 221 kB/s | 160 kB 00:00 2026-02-22T22:22:27.857 INFO:teuthology.orchestra.run.trial123.stdout:(20/35): librabbitmq-0.11.0-7.el9.x86_64.rpm 75 kB/s | 45 kB 00:00 2026-02-22T22:22:28.178 INFO:teuthology.orchestra.run.trial123.stdout:(21/35): librdkafka-1.6.1-102.el9.x86_64.rpm 757 kB/s | 662 kB 00:00 2026-02-22T22:22:28.296 INFO:teuthology.orchestra.run.trial123.stdout:(22/35): lttng-ust-2.12.0-6.el9.x86_64.rpm 367 kB/s | 292 kB 00:00 2026-02-22T22:22:28.613 INFO:teuthology.orchestra.run.trial123.stdout:(23/35): qatlib-25.08.0-2.el9.x86_64.rpm 317 kB/s | 240 kB 00:00 2026-02-22T22:22:28.747 INFO:teuthology.orchestra.run.trial123.stdout:(24/35): qatlib-service-25.08.0-2.el9.x86_64.rp 65 kB/s | 37 kB 00:00 2026-02-22T22:22:28.872 INFO:teuthology.orchestra.run.trial123.stdout:(25/35): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.2 MB/s | 308 kB 00:00 2026-02-22T22:22:28.918 INFO:teuthology.orchestra.run.trial123.stdout:(26/35): qatzip-libs-1.3.1-1.el9.x86_64.rpm 107 kB/s | 66 kB 00:00 2026-02-22T22:22:29.094 INFO:teuthology.orchestra.run.trial123.stdout:(27/35): libarrow-9.0.0-15.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2026-02-22T22:22:29.099 INFO:teuthology.orchestra.run.trial123.stdout:(28/35): libarrow-doc-9.0.0-15.el9.noarch.rpm 109 kB/s | 25 kB 00:00 2026-02-22T22:22:29.101 INFO:teuthology.orchestra.run.trial123.stdout:(29/35): liboath-2.6.12-1.el9.x86_64.rpm 268 kB/s | 49 kB 00:00 2026-02-22T22:22:29.128 INFO:teuthology.orchestra.run.trial123.stdout:(30/35): libunwind-1.6.2-1.el9.x86_64.rpm 1.9 MB/s | 67 kB 00:00 2026-02-22T22:22:29.189 INFO:teuthology.orchestra.run.trial123.stdout:(31/35): parquet-libs-9.0.0-15.el9.x86_64.rpm 9.1 MB/s | 838 kB 00:00 2026-02-22T22:22:29.199 INFO:teuthology.orchestra.run.trial123.stdout:(32/35): re2-20211101-20.el9.x86_64.rpm 1.9 MB/s | 191 kB 00:00 2026-02-22T22:22:29.300 INFO:teuthology.orchestra.run.trial123.stdout:(33/35): thrift-0.15.0-4.el9.x86_64.rpm 9.2 MB/s | 1.6 MB 00:00 2026-02-22T22:22:29.348 INFO:teuthology.orchestra.run.trial123.stdout:(34/35): librados2-19.2.3-689.gf90f4eae.el9.x86 22 MB/s | 3.4 MB 00:00 2026-02-22T22:22:29.352 INFO:teuthology.orchestra.run.trial123.stdout:(35/35): librbd1-19.2.3-689.gf90f4eae.el9.x86_6 21 MB/s | 3.2 MB 00:00 2026-02-22T22:22:29.354 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:22:29.354 INFO:teuthology.orchestra.run.trial123.stdout:Total 17 MB/s | 63 MB 00:03 2026-02-22T22:22:29.447 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:22:29.466 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:22:29.466 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:22:29.721 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:22:29.721 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:22:30.012 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:22:30.021 INFO:teuthology.orchestra.run.trial123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/37 2026-02-22T22:22:30.024 INFO:teuthology.orchestra.run.trial123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/37 2026-02-22T22:22:30.035 INFO:teuthology.orchestra.run.trial123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/37 2026-02-22T22:22:30.184 INFO:teuthology.orchestra.run.trial123.stdout: Installing : thrift-0.15.0-4.el9.x86_64 4/37 2026-02-22T22:22:30.188 INFO:teuthology.orchestra.run.trial123.stdout: Upgrading : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T22:22:30.219 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T22:22:30.261 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T22:22:30.262 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T22:22:30.274 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T22:22:30.278 INFO:teuthology.orchestra.run.trial123.stdout: Installing : liboath-2.6.12-1.el9.x86_64 8/37 2026-02-22T22:22:30.287 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 9/37 2026-02-22T22:22:30.309 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T22:22:30.309 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T22:22:30.322 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T22:22:30.358 INFO:teuthology.orchestra.run.trial123.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/37 2026-02-22T22:22:30.363 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 13/37 2026-02-22T22:22:30.369 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/37 2026-02-22T22:22:30.371 INFO:teuthology.orchestra.run.trial123.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/37 2026-02-22T22:22:30.374 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 16/37 2026-02-22T22:22:30.380 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/37 2026-02-22T22:22:30.520 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T22:22:30.521 INFO:teuthology.orchestra.run.trial123.stdout: Upgrading : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T22:22:30.546 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 19/37 2026-02-22T22:22:30.549 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 20/37 2026-02-22T22:22:30.555 INFO:teuthology.orchestra.run.trial123.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 21/37 2026-02-22T22:22:30.779 INFO:teuthology.orchestra.run.trial123.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 22/37 2026-02-22T22:22:30.781 INFO:teuthology.orchestra.run.trial123.stdout: Installing : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T22:22:30.796 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 23/37 2026-02-22T22:22:30.802 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 24/37 2026-02-22T22:22:30.808 INFO:teuthology.orchestra.run.trial123.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 25/37 2026-02-22T22:22:30.810 INFO:teuthology.orchestra.run.trial123.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 26/37 2026-02-22T22:22:30.811 INFO:teuthology.orchestra.run.trial123.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T22:22:30.829 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 27/37 2026-02-22T22:22:30.947 INFO:teuthology.orchestra.run.trial123.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 28/37 2026-02-22T22:22:30.950 INFO:teuthology.orchestra.run.trial123.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T22:22:31.245 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 29/37 2026-02-22T22:22:31.245 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T22:22:31.245 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:31.282 INFO:teuthology.orchestra.run.trial123.stdout: Installing : mailcap-2.1.49-5.el9.noarch 30/37 2026-02-22T22:22:31.284 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 31/37 2026-02-22T22:22:32.262 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T22:22:32.265 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T22:22:32.545 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 32/37 2026-02-22T22:22:32.548 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T22:22:32.589 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 33/37 2026-02-22T22:22:32.589 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-22T22:22:32.589 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-22T22:22:32.590 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:32.592 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T22:22:37.141 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T22:22:37.141 INFO:teuthology.orchestra.run.trial123.stdout:skipping the directory /sys 2026-02-22T22:22:37.141 INFO:teuthology.orchestra.run.trial123.stdout:skipping the directory /proc 2026-02-22T22:22:37.141 INFO:teuthology.orchestra.run.trial123.stdout:skipping the directory /mnt 2026-02-22T22:22:37.141 INFO:teuthology.orchestra.run.trial123.stdout:skipping the directory /var/tmp 2026-02-22T22:22:37.142 INFO:teuthology.orchestra.run.trial123.stdout:skipping the directory /home 2026-02-22T22:22:37.142 INFO:teuthology.orchestra.run.trial123.stdout:skipping the directory /root 2026-02-22T22:22:37.142 INFO:teuthology.orchestra.run.trial123.stdout:skipping the directory /tmp 2026-02-22T22:22:37.142 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:37.588 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T22:22:37.610 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 35/37 2026-02-22T22:22:37.610 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:22:37.610 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-22T22:22:37.610 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T22:22:37.610 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-22T22:22:37.611 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:37.612 INFO:teuthology.orchestra.run.trial123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T22:22:37.623 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 36/37 2026-02-22T22:22:37.623 INFO:teuthology.orchestra.run.trial123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T22:22:37.678 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T22:22:37.678 INFO:teuthology.orchestra.run.trial123.stdout:Creating group 'qat' with GID 992. 2026-02-22T22:22:37.678 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:37.958 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 1/37 2026-02-22T22:22:37.958 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2/37 2026-02-22T22:22:37.958 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 3/37 2026-02-22T22:22:37.958 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 4/37 2026-02-22T22:22:37.958 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 5/37 2026-02-22T22:22:37.958 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 6/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 7/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x 8/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86 9/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 10/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 11/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 12/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 13/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 14/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 15/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 16/37 2026-02-22T22:22:37.959 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 17/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 18/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 19/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 20/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 21/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 22/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 23/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 24/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 25/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 26/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 27/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 28/37 2026-02-22T22:22:37.960 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 29/37 2026-02-22T22:22:37.961 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 30/37 2026-02-22T22:22:37.961 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 31/37 2026-02-22T22:22:37.961 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : re2-1:20211101-20.el9.x86_64 32/37 2026-02-22T22:22:37.961 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 33/37 2026-02-22T22:22:37.961 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 34/37 2026-02-22T22:22:37.961 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 35/37 2026-02-22T22:22:37.961 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 36/37 2026-02-22T22:22:38.019 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 37/37 2026-02-22T22:22:38.019 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:38.019 INFO:teuthology.orchestra.run.trial123.stdout:Upgraded: 2026-02-22T22:22:38.019 INFO:teuthology.orchestra.run.trial123.stdout: librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.019 INFO:teuthology.orchestra.run.trial123.stdout: librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.019 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:22:38.019 INFO:teuthology.orchestra.run.trial123.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: ceph-radosgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: ceph-selinux-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-22T22:22:38.020 INFO:teuthology.orchestra.run.trial123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: libradosstriper1-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: librgw2-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: python3-ceph-argparse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.021 INFO:teuthology.orchestra.run.trial123.stdout: python3-ceph-common-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: re2-1:20211101-20.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:38.022 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:22:38.062 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-test 2026-02-22T22:22:38.375 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:14 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:22:38.458 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:22:38.458 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:22:38.458 INFO:teuthology.orchestra.run.trial123.stdout: Package Arch Version Repository Size 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout: ceph-test x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 50 M 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout:Installing dependencies: 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout: libcephsqlite x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 163 k 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:22:38.459 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:22:38.460 INFO:teuthology.orchestra.run.trial123.stdout:Install 5 Packages 2026-02-22T22:22:38.460 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:38.460 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 50 M 2026-02-22T22:22:38.460 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 212 M 2026-02-22T22:22:38.461 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:22:38.881 INFO:teuthology.orchestra.run.trial123.stdout:(1/5): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-22T22:22:38.933 INFO:teuthology.orchestra.run.trial123.stdout:(2/5): socat-1.7.4.1-8.el9.x86_64.rpm 5.6 MB/s | 303 kB 00:00 2026-02-22T22:22:38.959 INFO:teuthology.orchestra.run.trial123.stdout:(3/5): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 64 kB 00:00 2026-02-22T22:22:39.125 INFO:teuthology.orchestra.run.trial123.stdout:(4/5): libcephsqlite-19.2.3-689.gf90f4eae.el9.x 401 kB/s | 163 kB 00:00 2026-02-22T22:22:41.121 INFO:teuthology.orchestra.run.trial123.stdout:(5/5): ceph-test-19.2.3-689.gf90f4eae.el9.x86_6 21 MB/s | 50 MB 00:02 2026-02-22T22:22:41.122 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:22:41.122 INFO:teuthology.orchestra.run.trial123.stdout:Total 19 MB/s | 50 MB 00:02 2026-02-22T22:22:41.149 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:22:41.157 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:22:41.157 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:22:41.343 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:22:41.343 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:22:41.554 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:22:41.559 INFO:teuthology.orchestra.run.trial123.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 1/5 2026-02-22T22:22:41.564 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 2/5 2026-02-22T22:22:41.572 INFO:teuthology.orchestra.run.trial123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/5 2026-02-22T22:22:41.572 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T22:22:43.960 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 4/5 2026-02-22T22:22:43.978 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T22:22:44.214 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 5/5 2026-02-22T22:22:44.214 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 1/5 2026-02-22T22:22:44.214 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2/5 2026-02-22T22:22:44.215 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 3/5 2026-02-22T22:22:44.215 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 4/5 2026-02-22T22:22:44.273 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/5 2026-02-22T22:22:44.273 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:44.273 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:22:44.274 INFO:teuthology.orchestra.run.trial123.stdout: ceph-test-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:44.274 INFO:teuthology.orchestra.run.trial123.stdout: libcephsqlite-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:22:44.274 INFO:teuthology.orchestra.run.trial123.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-22T22:22:44.274 INFO:teuthology.orchestra.run.trial123.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-22T22:22:44.274 INFO:teuthology.orchestra.run.trial123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-22T22:22:44.274 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:44.274 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:22:44.308 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph 2026-02-22T22:22:44.615 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:20 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:22:44.702 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:22:44.705 INFO:teuthology.orchestra.run.trial123.stdout:====================================================================================== 2026-02-22T22:22:44.705 INFO:teuthology.orchestra.run.trial123.stdout: Package Arch Version Repository Size 2026-02-22T22:22:44.705 INFO:teuthology.orchestra.run.trial123.stdout:====================================================================================== 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: ceph x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 6.5 k 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout:Installing dependencies: 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: ceph-grafana-dashboards noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 31 k 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mds x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 2.4 M 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.1 M 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-modules-core noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 253 k 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mon x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 4.7 M 2026-02-22T22:22:44.706 INFO:teuthology.orchestra.run.trial123.stdout: ceph-osd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 17 M 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: ceph-prometheus-alerts noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 17 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 769 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-22T22:22:44.707 INFO:teuthology.orchestra.run.trial123.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-22T22:22:44.708 INFO:teuthology.orchestra.run.trial123.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-22T22:22:44.709 INFO:teuthology.orchestra.run.trial123.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2026-02-22T22:22:44.710 INFO:teuthology.orchestra.run.trial123.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-22T22:22:44.711 INFO:teuthology.orchestra.run.trial123.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-22T22:22:44.712 INFO:teuthology.orchestra.run.trial123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-22T22:22:44.713 INFO:teuthology.orchestra.run.trial123.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout:Installing weak dependencies: 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-cephadm noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 150 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-dashboard noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 3.8 M 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-diskprediction-local noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 7.4 M 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-k8sevents noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 22 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-rook noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 49 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: ceph-volume noarch 2:19.2.3-689.gf90f4eae.el9 ceph-noarch 299 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-22T22:22:44.714 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:44.715 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:22:44.715 INFO:teuthology.orchestra.run.trial123.stdout:====================================================================================== 2026-02-22T22:22:44.715 INFO:teuthology.orchestra.run.trial123.stdout:Install 99 Packages 2026-02-22T22:22:44.715 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:22:44.716 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 95 M 2026-02-22T22:22:44.716 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 520 M 2026-02-22T22:22:44.716 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:22:45.058 INFO:teuthology.orchestra.run.trial123.stdout:(1/99): ceph-19.2.3-689.gf90f4eae.el9.x86_64.rp 126 kB/s | 6.5 kB 00:00 2026-02-22T22:22:45.330 INFO:teuthology.orchestra.run.trial123.stdout:(2/99): ceph-mgr-19.2.3-689.gf90f4eae.el9.x86_6 3.3 MB/s | 1.1 MB 00:00 2026-02-22T22:22:45.453 INFO:teuthology.orchestra.run.trial123.stdout:(3/99): ceph-mds-19.2.3-689.gf90f4eae.el9.x86_6 5.4 MB/s | 2.4 MB 00:00 2026-02-22T22:22:45.486 INFO:teuthology.orchestra.run.trial123.stdout:(4/99): ceph-grafana-dashboards-19.2.3-689.gf90 937 kB/s | 31 kB 00:00 2026-02-22T22:22:45.523 INFO:teuthology.orchestra.run.trial123.stdout:(5/99): ceph-mgr-cephadm-19.2.3-689.gf90f4eae.e 3.9 MB/s | 150 kB 00:00 2026-02-22T22:22:45.760 INFO:teuthology.orchestra.run.trial123.stdout:(6/99): ceph-mon-19.2.3-689.gf90f4eae.el9.x86_6 6.7 MB/s | 4.7 MB 00:00 2026-02-22T22:22:46.137 INFO:teuthology.orchestra.run.trial123.stdout:(7/99): ceph-mgr-dashboard-19.2.3-689.gf90f4eae 6.2 MB/s | 3.8 MB 00:00 2026-02-22T22:22:46.157 INFO:teuthology.orchestra.run.trial123.stdout:(8/99): ceph-mgr-k8sevents-19.2.3-689.gf90f4eae 1.0 MB/s | 22 kB 00:00 2026-02-22T22:22:46.182 INFO:teuthology.orchestra.run.trial123.stdout:(9/99): ceph-mgr-modules-core-19.2.3-689.gf90f4 10 MB/s | 253 kB 00:00 2026-02-22T22:22:46.204 INFO:teuthology.orchestra.run.trial123.stdout:(10/99): ceph-mgr-rook-19.2.3-689.gf90f4eae.el9 2.3 MB/s | 49 kB 00:00 2026-02-22T22:22:46.225 INFO:teuthology.orchestra.run.trial123.stdout:(11/99): ceph-prometheus-alerts-19.2.3-689.gf90 782 kB/s | 17 kB 00:00 2026-02-22T22:22:46.252 INFO:teuthology.orchestra.run.trial123.stdout:(12/99): ceph-volume-19.2.3-689.gf90f4eae.el9.n 11 MB/s | 299 kB 00:00 2026-02-22T22:22:46.312 INFO:teuthology.orchestra.run.trial123.stdout:(13/99): cephadm-19.2.3-689.gf90f4eae.el9.noarc 13 MB/s | 769 kB 00:00 2026-02-22T22:22:46.346 INFO:teuthology.orchestra.run.trial123.stdout:(14/99): ceph-mgr-diskprediction-local-19.2.3-6 13 MB/s | 7.4 MB 00:00 2026-02-22T22:22:46.609 INFO:teuthology.orchestra.run.trial123.stdout:(15/99): ceph-osd-19.2.3-689.gf90f4eae.el9.x86_ 13 MB/s | 17 MB 00:01 2026-02-22T22:22:46.623 INFO:teuthology.orchestra.run.trial123.stdout:(16/99): ledmon-libs-1.1.0-3.el9.x86_64.rpm 146 kB/s | 40 kB 00:00 2026-02-22T22:22:46.856 INFO:teuthology.orchestra.run.trial123.stdout:(17/99): cryptsetup-2.8.1-3.el9.x86_64.rpm 646 kB/s | 351 kB 00:00 2026-02-22T22:22:46.963 INFO:teuthology.orchestra.run.trial123.stdout:(18/99): libquadmath-11.5.0-14.el9.x86_64.rpm 1.7 MB/s | 184 kB 00:00 2026-02-22T22:22:46.966 INFO:teuthology.orchestra.run.trial123.stdout:(19/99): libconfig-1.7.2-9.el9.x86_64.rpm 201 kB/s | 72 kB 00:00 2026-02-22T22:22:47.058 INFO:teuthology.orchestra.run.trial123.stdout:(20/99): libgfortran-11.5.0-14.el9.x86_64.rpm 1.8 MB/s | 794 kB 00:00 2026-02-22T22:22:47.081 INFO:teuthology.orchestra.run.trial123.stdout:(21/99): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2026-02-22T22:22:47.143 INFO:teuthology.orchestra.run.trial123.stdout:(22/99): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2026-02-22T22:22:47.180 INFO:teuthology.orchestra.run.trial123.stdout:(23/99): python3-pycparser-2.20-6.el9.noarch.rp 1.3 MB/s | 135 kB 00:00 2026-02-22T22:22:47.230 INFO:teuthology.orchestra.run.trial123.stdout:(24/99): python3-pyparsing-2.4.7-9.el9.noarch.r 1.7 MB/s | 150 kB 00:00 2026-02-22T22:22:47.278 INFO:teuthology.orchestra.run.trial123.stdout:(25/99): python3-requests-2.25.1-10.el9.noarch. 1.3 MB/s | 126 kB 00:00 2026-02-22T22:22:47.322 INFO:teuthology.orchestra.run.trial123.stdout:(26/99): python3-urllib3-1.26.5-7.el9.noarch.rp 2.3 MB/s | 218 kB 00:00 2026-02-22T22:22:47.383 INFO:teuthology.orchestra.run.trial123.stdout:(27/99): unzip-6.0-59.el9.x86_64.rpm 1.7 MB/s | 182 kB 00:00 2026-02-22T22:22:47.394 INFO:teuthology.orchestra.run.trial123.stdout:(28/99): python3-cryptography-36.0.1-5.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2026-02-22T22:22:47.416 INFO:teuthology.orchestra.run.trial123.stdout:(29/99): zip-3.0-35.el9.x86_64.rpm 2.7 MB/s | 266 kB 00:00 2026-02-22T22:22:47.787 INFO:teuthology.orchestra.run.trial123.stdout:(30/99): flexiblas-3.0.4-9.el9.x86_64.rpm 73 kB/s | 30 kB 00:00 2026-02-22T22:22:47.818 INFO:teuthology.orchestra.run.trial123.stdout:(31/99): flexiblas-openblas-openmp-3.0.4-9.el9. 37 kB/s | 15 kB 00:00 2026-02-22T22:22:48.135 INFO:teuthology.orchestra.run.trial123.stdout:(32/99): lua-5.4.4-4.el9.x86_64.rpm 594 kB/s | 188 kB 00:00 2026-02-22T22:22:48.181 INFO:teuthology.orchestra.run.trial123.stdout:(33/99): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 626 kB/s | 246 kB 00:00 2026-02-22T22:22:48.219 INFO:teuthology.orchestra.run.trial123.stdout:(34/99): openblas-0.3.29-1.el9.x86_64.rpm 506 kB/s | 42 kB 00:00 2026-02-22T22:22:49.395 INFO:teuthology.orchestra.run.trial123.stdout:(35/99): protobuf-3.14.0-17.el9.x86_64.rpm 874 kB/s | 1.0 MB 00:01 2026-02-22T22:22:51.264 INFO:teuthology.orchestra.run.trial123.stdout:(36/99): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 791 kB/s | 3.0 MB 00:03 2026-02-22T22:22:51.586 INFO:teuthology.orchestra.run.trial123.stdout:(37/99): python3-devel-3.9.25-3.el9.x86_64.rpm 759 kB/s | 244 kB 00:00 2026-02-22T22:22:51.907 INFO:teuthology.orchestra.run.trial123.stdout:(38/99): python3-jinja2-2.11.3-8.el9.noarch.rpm 775 kB/s | 249 kB 00:00 2026-02-22T22:22:52.148 INFO:teuthology.orchestra.run.trial123.stdout:(39/99): python3-libstoragemgmt-1.10.1-1.el9.x8 732 kB/s | 177 kB 00:00 2026-02-22T22:22:53.573 INFO:teuthology.orchestra.run.trial123.stdout:(40/99): python3-lxml-4.6.5-3.el9.x86_64.rpm 879 kB/s | 1.2 MB 00:01 2026-02-22T22:22:53.815 INFO:teuthology.orchestra.run.trial123.stdout:(41/99): python3-mako-1.1.4-6.el9.noarch.rpm 710 kB/s | 172 kB 00:00 2026-02-22T22:22:53.900 INFO:teuthology.orchestra.run.trial123.stdout:(42/99): python3-markupsafe-1.1.1-12.el9.x86_64 413 kB/s | 35 kB 00:00 2026-02-22T22:22:54.176 INFO:teuthology.orchestra.run.trial123.stdout:(43/99): openblas-openmp-0.3.29-1.el9.x86_64.rp 903 kB/s | 5.3 MB 00:05 2026-02-22T22:22:54.727 INFO:teuthology.orchestra.run.trial123.stdout:(44/99): python3-numpy-f2py-1.23.5-2.el9.x86_64 802 kB/s | 442 kB 00:00 2026-02-22T22:22:54.811 INFO:teuthology.orchestra.run.trial123.stdout:[MIRROR] python3-packaging-20.9-5.el9.noarch.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-packaging-20.9-5.el9.noarch.rpm (IP: 140.211.166.134) 2026-02-22T22:22:55.467 INFO:teuthology.orchestra.run.trial123.stdout:(45/99): python3-packaging-20.9-5.el9.noarch.rp 104 kB/s | 77 kB 00:00 2026-02-22T22:22:56.195 INFO:teuthology.orchestra.run.trial123.stdout:(46/99): python3-babel-2.9.1-2.el9.noarch.rpm 898 kB/s | 6.0 MB 00:06 2026-02-22T22:22:56.357 INFO:teuthology.orchestra.run.trial123.stdout:(47/99): python3-pyasn1-0.4.8-6.el9.noarch.rpm 978 kB/s | 159 kB 00:00 2026-02-22T22:22:56.442 INFO:teuthology.orchestra.run.trial123.stdout:(48/99): python3-pyasn1-modules-0.4.8-6.el9.noa 3.2 MB/s | 279 kB 00:00 2026-02-22T22:22:56.492 INFO:teuthology.orchestra.run.trial123.stdout:(49/99): python3-protobuf-3.14.0-17.el9.noarch. 261 kB/s | 267 kB 00:01 2026-02-22T22:22:56.526 INFO:teuthology.orchestra.run.trial123.stdout:(50/99): python3-requests-oauthlib-1.3.0-12.el9 642 kB/s | 54 kB 00:00 2026-02-22T22:22:56.609 INFO:teuthology.orchestra.run.trial123.stdout:(51/99): python3-toml-0.10.2-6.el9.noarch.rpm 500 kB/s | 42 kB 00:00 2026-02-22T22:22:56.693 INFO:teuthology.orchestra.run.trial123.stdout:(52/99): xmlsec1-1.2.29-13.el9.x86_64.rpm 2.2 MB/s | 189 kB 00:00 2026-02-22T22:22:56.777 INFO:teuthology.orchestra.run.trial123.stdout:(53/99): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-22T22:22:56.860 INFO:teuthology.orchestra.run.trial123.stdout:(54/99): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2026-02-22T22:22:57.022 INFO:teuthology.orchestra.run.trial123.stdout:(55/99): protobuf-compiler-3.14.0-17.el9.x86_64 5.2 MB/s | 862 kB 00:00 2026-02-22T22:22:57.152 INFO:teuthology.orchestra.run.trial123.stdout:(56/99): abseil-cpp-20211102.0-4.el9.x86_64.rpm 4.2 MB/s | 551 kB 00:00 2026-02-22T22:22:57.190 INFO:teuthology.orchestra.run.trial123.stdout:(57/99): grpc-data-1.46.7-10.el9.noarch.rpm 505 kB/s | 19 kB 00:00 2026-02-22T22:22:57.223 INFO:teuthology.orchestra.run.trial123.stdout:(58/99): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2026-02-22T22:22:57.265 INFO:teuthology.orchestra.run.trial123.stdout:(59/99): python3-asyncssh-2.13.2-5.el9.noarch.r 13 MB/s | 548 kB 00:00 2026-02-22T22:22:57.296 INFO:teuthology.orchestra.run.trial123.stdout:(60/99): python3-autocommand-2.2.2-8.el9.noarch 962 kB/s | 29 kB 00:00 2026-02-22T22:22:57.334 INFO:teuthology.orchestra.run.trial123.stdout:(61/99): python3-backports-tarfile-1.2.0-1.el9. 1.5 MB/s | 60 kB 00:00 2026-02-22T22:22:57.368 INFO:teuthology.orchestra.run.trial123.stdout:(62/99): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2026-02-22T22:22:57.400 INFO:teuthology.orchestra.run.trial123.stdout:(63/99): python3-cachetools-4.2.4-1.el9.noarch. 1.0 MB/s | 32 kB 00:00 2026-02-22T22:22:57.431 INFO:teuthology.orchestra.run.trial123.stdout:(64/99): python3-certifi-2023.05.07-4.el9.noarc 448 kB/s | 14 kB 00:00 2026-02-22T22:22:57.468 INFO:teuthology.orchestra.run.trial123.stdout:(65/99): python3-cheroot-10.0.1-4.el9.noarch.rp 4.7 MB/s | 173 kB 00:00 2026-02-22T22:22:57.506 INFO:teuthology.orchestra.run.trial123.stdout:(66/99): python3-cherrypy-18.6.1-2.el9.noarch.r 9.3 MB/s | 358 kB 00:00 2026-02-22T22:22:57.544 INFO:teuthology.orchestra.run.trial123.stdout:(67/99): python3-google-auth-2.45.0-1.el9.noarc 6.4 MB/s | 254 kB 00:00 2026-02-22T22:22:57.624 INFO:teuthology.orchestra.run.trial123.stdout:(68/99): python3-grpcio-1.46.7-10.el9.x86_64.rp 26 MB/s | 2.0 MB 00:00 2026-02-22T22:22:57.650 INFO:teuthology.orchestra.run.trial123.stdout:(69/99): python3-grpcio-tools-1.46.7-10.el9.x86 5.3 MB/s | 144 kB 00:00 2026-02-22T22:22:57.681 INFO:teuthology.orchestra.run.trial123.stdout:(70/99): python3-influxdb-5.3.1-1.el9.noarch.rp 4.4 MB/s | 139 kB 00:00 2026-02-22T22:22:57.710 INFO:teuthology.orchestra.run.trial123.stdout:(71/99): python3-isodate-0.6.1-3.el9.noarch.rpm 1.9 MB/s | 56 kB 00:00 2026-02-22T22:22:57.740 INFO:teuthology.orchestra.run.trial123.stdout:(72/99): python3-jaraco-8.2.1-3.el9.noarch.rpm 356 kB/s | 11 kB 00:00 2026-02-22T22:22:57.767 INFO:teuthology.orchestra.run.trial123.stdout:(73/99): python3-jaraco-classes-3.2.1-5.el9.noa 656 kB/s | 18 kB 00:00 2026-02-22T22:22:57.802 INFO:teuthology.orchestra.run.trial123.stdout:(74/99): python3-jaraco-collections-3.0.0-8.el9 667 kB/s | 23 kB 00:00 2026-02-22T22:22:57.841 INFO:teuthology.orchestra.run.trial123.stdout:(75/99): python3-jaraco-context-6.0.1-3.el9.noa 508 kB/s | 20 kB 00:00 2026-02-22T22:22:57.873 INFO:teuthology.orchestra.run.trial123.stdout:(76/99): python3-jaraco-functools-3.5.0-2.el9.n 604 kB/s | 19 kB 00:00 2026-02-22T22:22:57.903 INFO:teuthology.orchestra.run.trial123.stdout:(77/99): python3-jaraco-text-4.0.0-2.el9.noarch 881 kB/s | 26 kB 00:00 2026-02-22T22:22:57.943 INFO:teuthology.orchestra.run.trial123.stdout:(78/99): python3-kubernetes-26.1.0-3.el9.noarch 26 MB/s | 1.0 MB 00:00 2026-02-22T22:22:57.975 INFO:teuthology.orchestra.run.trial123.stdout:(79/99): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2026-02-22T22:22:58.004 INFO:teuthology.orchestra.run.trial123.stdout:(80/99): python3-more-itertools-8.12.0-2.el9.no 2.7 MB/s | 79 kB 00:00 2026-02-22T22:22:58.042 INFO:teuthology.orchestra.run.trial123.stdout:(81/99): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.2 MB/s | 86 kB 00:00 2026-02-22T22:22:58.075 INFO:teuthology.orchestra.run.trial123.stdout:(82/99): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2026-02-22T22:22:58.114 INFO:teuthology.orchestra.run.trial123.stdout:(83/99): python3-pecan-1.4.2-3.el9.noarch.rpm 6.9 MB/s | 272 kB 00:00 2026-02-22T22:22:58.146 INFO:teuthology.orchestra.run.trial123.stdout:(84/99): python3-portend-3.1.0-2.el9.noarch.rpm 515 kB/s | 16 kB 00:00 2026-02-22T22:22:58.184 INFO:teuthology.orchestra.run.trial123.stdout:(85/99): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.3 MB/s | 90 kB 00:00 2026-02-22T22:22:58.216 INFO:teuthology.orchestra.run.trial123.stdout:(86/99): python3-repoze-lru-0.7-16.el9.noarch.r 963 kB/s | 31 kB 00:00 2026-02-22T22:22:58.246 INFO:teuthology.orchestra.run.trial123.stdout:(87/99): python3-routes-2.5.1-5.el9.noarch.rpm 6.0 MB/s | 188 kB 00:00 2026-02-22T22:22:58.279 INFO:teuthology.orchestra.run.trial123.stdout:(88/99): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2026-02-22T22:22:58.310 INFO:teuthology.orchestra.run.trial123.stdout:(89/99): python3-saml-1.16.0-1.el9.noarch.rpm 4.0 MB/s | 125 kB 00:00 2026-02-22T22:22:58.342 INFO:teuthology.orchestra.run.trial123.stdout:(90/99): python3-tempora-5.0.0-2.el9.noarch.rpm 1.1 MB/s | 36 kB 00:00 2026-02-22T22:22:58.377 INFO:teuthology.orchestra.run.trial123.stdout:(91/99): python3-typing-extensions-4.15.0-1.el9 2.4 MB/s | 86 kB 00:00 2026-02-22T22:22:58.417 INFO:teuthology.orchestra.run.trial123.stdout:(92/99): python3-webob-1.8.8-2.el9.noarch.rpm 5.7 MB/s | 230 kB 00:00 2026-02-22T22:22:58.449 INFO:teuthology.orchestra.run.trial123.stdout:(93/99): python3-websocket-client-1.2.3-2.el9.n 2.8 MB/s | 90 kB 00:00 2026-02-22T22:22:58.493 INFO:teuthology.orchestra.run.trial123.stdout:(94/99): python3-werkzeug-2.0.3-3.el9.1.noarch. 9.4 MB/s | 427 kB 00:00 2026-02-22T22:22:58.523 INFO:teuthology.orchestra.run.trial123.stdout:(95/99): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.6 MB/s | 48 kB 00:00 2026-02-22T22:22:58.554 INFO:teuthology.orchestra.run.trial123.stdout:(96/99): python3-xmltodict-0.12.0-15.el9.noarch 718 kB/s | 22 kB 00:00 2026-02-22T22:22:58.590 INFO:teuthology.orchestra.run.trial123.stdout:(97/99): python3-zc-lockfile-2.0-10.el9.noarch. 563 kB/s | 20 kB 00:00 2026-02-22T22:22:59.272 INFO:teuthology.orchestra.run.trial123.stdout:(98/99): python3-numpy-1.23.5-2.el9.x86_64.rpm 1.1 MB/s | 6.1 MB 00:05 2026-02-22T22:23:05.962 INFO:teuthology.orchestra.run.trial123.stdout:(99/99): python3-scipy-1.9.3-2.el9.x86_64.rpm 2.0 MB/s | 19 MB 00:09 2026-02-22T22:23:05.965 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:05.965 INFO:teuthology.orchestra.run.trial123.stdout:Total 4.5 MB/s | 95 MB 00:21 2026-02-22T22:23:06.389 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:06.427 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:06.428 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:06.845 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:06.845 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:07.310 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:07.312 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/99 2026-02-22T22:23:07.343 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/99 2026-02-22T22:23:07.348 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 3/99 2026-02-22T22:23:07.370 INFO:teuthology.orchestra.run.trial123.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 4/99 2026-02-22T22:23:07.376 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 5/99 2026-02-22T22:23:07.410 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 6/99 2026-02-22T22:23:07.419 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 7/99 2026-02-22T22:23:07.424 INFO:teuthology.orchestra.run.trial123.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 8/99 2026-02-22T22:23:07.427 INFO:teuthology.orchestra.run.trial123.stdout: Installing : lua-5.4.4-4.el9.x86_64 9/99 2026-02-22T22:23:07.431 INFO:teuthology.orchestra.run.trial123.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 10/99 2026-02-22T22:23:07.455 INFO:teuthology.orchestra.run.trial123.stdout: Installing : unzip-6.0-59.el9.x86_64 11/99 2026-02-22T22:23:07.467 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 12/99 2026-02-22T22:23:07.470 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 13/99 2026-02-22T22:23:07.476 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 14/99 2026-02-22T22:23:07.478 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 15/99 2026-02-22T22:23:07.491 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 16/99 2026-02-22T22:23:07.497 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 17/99 2026-02-22T22:23:07.523 INFO:teuthology.orchestra.run.trial123.stdout: Installing : zip-3.0-35.el9.x86_64 18/99 2026-02-22T22:23:07.526 INFO:teuthology.orchestra.run.trial123.stdout: Installing : luarocks-3.9.2-5.el9.noarch 19/99 2026-02-22T22:23:07.533 INFO:teuthology.orchestra.run.trial123.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 20/99 2026-02-22T22:23:07.558 INFO:teuthology.orchestra.run.trial123.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 21/99 2026-02-22T22:23:07.610 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 22/99 2026-02-22T22:23:07.623 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 23/99 2026-02-22T22:23:07.627 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 24/99 2026-02-22T22:23:07.632 INFO:teuthology.orchestra.run.trial123.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 25/99 2026-02-22T22:23:07.637 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 26/99 2026-02-22T22:23:07.642 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 27/99 2026-02-22T22:23:07.645 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 28/99 2026-02-22T22:23:07.660 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 29/99 2026-02-22T22:23:07.681 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 30/99 2026-02-22T22:23:07.686 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 31/99 2026-02-22T22:23:07.691 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 32/99 2026-02-22T22:23:07.702 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 33/99 2026-02-22T22:23:07.711 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 34/99 2026-02-22T22:23:07.717 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 35/99 2026-02-22T22:23:07.737 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 36/99 2026-02-22T22:23:07.746 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 37/99 2026-02-22T22:23:07.801 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 38/99 2026-02-22T22:23:07.813 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 39/99 2026-02-22T22:23:07.829 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 40/99 2026-02-22T22:23:07.834 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 41/99 2026-02-22T22:23:07.842 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 42/99 2026-02-22T22:23:07.883 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 43/99 2026-02-22T22:23:08.211 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 44/99 2026-02-22T22:23:08.221 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 45/99 2026-02-22T22:23:08.225 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 46/99 2026-02-22T22:23:08.231 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 47/99 2026-02-22T22:23:08.233 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 48/99 2026-02-22T22:23:08.261 INFO:teuthology.orchestra.run.trial123.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 49/99 2026-02-22T22:23:08.305 INFO:teuthology.orchestra.run.trial123.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 50/99 2026-02-22T22:23:08.312 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 51/99 2026-02-22T22:23:08.315 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 52/99 2026-02-22T22:23:08.321 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 53/99 2026-02-22T22:23:08.325 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 54/99 2026-02-22T22:23:08.332 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 55/99 2026-02-22T22:23:08.335 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 56/99 2026-02-22T22:23:08.363 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 57/99 2026-02-22T22:23:08.373 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 58/99 2026-02-22T22:23:08.408 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 59/99 2026-02-22T22:23:08.634 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 60/99 2026-02-22T22:23:08.658 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 61/99 2026-02-22T22:23:08.662 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 62/99 2026-02-22T22:23:08.720 INFO:teuthology.orchestra.run.trial123.stdout: Installing : openblas-0.3.29-1.el9.x86_64 63/99 2026-02-22T22:23:08.722 INFO:teuthology.orchestra.run.trial123.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 64/99 2026-02-22T22:23:08.743 INFO:teuthology.orchestra.run.trial123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 65/99 2026-02-22T22:23:09.072 INFO:teuthology.orchestra.run.trial123.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 66/99 2026-02-22T22:23:09.145 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 67/99 2026-02-22T22:23:09.823 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 68/99 2026-02-22T22:23:09.840 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 69/99 2026-02-22T22:23:09.850 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 70/99 2026-02-22T22:23:09.857 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-packaging-20.9-5.el9.noarch 71/99 2026-02-22T22:23:09.871 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-ply-3.11-14.el9.noarch 72/99 2026-02-22T22:23:09.887 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 73/99 2026-02-22T22:23:09.963 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 74/99 2026-02-22T22:23:09.971 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 75/99 2026-02-22T22:23:09.995 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 76/99 2026-02-22T22:23:10.025 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 77/99 2026-02-22T22:23:10.077 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/99 2026-02-22T22:23:10.083 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 79/99 2026-02-22T22:23:10.086 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 80/99 2026-02-22T22:23:10.087 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 81/99 2026-02-22T22:23:10.106 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T22:23:10.106 INFO:teuthology.orchestra.run.trial123.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-22T22:23:10.106 INFO:teuthology.orchestra.run.trial123.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-22T22:23:10.106 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:10.115 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T22:23:10.142 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 82/99 2026-02-22T22:23:10.142 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-22T22:23:10.142 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:10.176 INFO:teuthology.orchestra.run.trial123.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 83/99 2026-02-22T22:23:10.241 INFO:teuthology.orchestra.run.trial123.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 84/99 2026-02-22T22:23:10.244 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T22:23:10.257 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 85/99 2026-02-22T22:23:10.258 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:23:10.258 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-22T22:23:10.258 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:10.954 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T22:23:10.979 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 86/99 2026-02-22T22:23:10.979 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:23:10.980 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-22T22:23:10.980 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T22:23:10.980 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-22T22:23:10.980 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:11.037 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T22:23:11.039 INFO:teuthology.orchestra.run.trial123.stdout: Installing : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 87/99 2026-02-22T22:23:11.044 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 88/99 2026-02-22T22:23:11.063 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 89/99 2026-02-22T22:23:11.065 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T22:23:11.557 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 90/99 2026-02-22T22:23:11.560 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T22:23:12.013 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 91/99 2026-02-22T22:23:12.013 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T22:23:12.028 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 92/99 2026-02-22T22:23:12.028 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T22:23:12.084 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 93/99 2026-02-22T22:23:12.133 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 94/99 2026-02-22T22:23:12.134 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T22:23:12.155 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 95/99 2026-02-22T22:23:12.155 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:23:12.155 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-22T22:23:12.155 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T22:23:12.155 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-22T22:23:12.155 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:12.167 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T22:23:12.383 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 96/99 2026-02-22T22:23:12.385 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T22:23:12.408 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 97/99 2026-02-22T22:23:12.408 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:23:12.408 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-22T22:23:12.408 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T22:23:12.408 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-22T22:23:12.408 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:12.509 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T22:23:12.530 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 98/99 2026-02-22T22:23:12.530 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:23:12.530 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-22T22:23:12.530 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T22:23:12.530 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-22T22:23:12.530 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:12.548 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T22:23:13.227 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 99/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 1/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 3/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 4/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 5/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9 6/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 7/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noar 8/99 2026-02-22T22:23:13.228 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4e 9/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noar 10/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.n 11/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 12/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9. 13/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 14/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 15/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 16/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 17/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 18/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 19/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 20/99 2026-02-22T22:23:13.229 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 21/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 22/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 23/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 24/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 25/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 26/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 27/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : unzip-6.0-59.el9.x86_64 28/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : zip-3.0-35.el9.x86_64 29/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 30/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 31/99 2026-02-22T22:23:13.230 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 32/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 33/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : lua-5.4.4-4.el9.x86_64 34/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 35/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 36/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 38/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 39/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 40/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 41/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 42/99 2026-02-22T22:23:13.231 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 43/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 44/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 45/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 46/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 47/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 48/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 49/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 50/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 51/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 52/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 53/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 54/99 2026-02-22T22:23:13.232 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 55/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 56/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 57/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 58/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 59/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 60/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 61/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 62/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 63/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 64/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 65/99 2026-02-22T22:23:13.233 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 66/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 67/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 68/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 69/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 70/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 71/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 72/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 73/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 74/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 75/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 76/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 77/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 78/99 2026-02-22T22:23:13.234 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 79/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 80/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 81/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 85/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 86/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 88/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 89/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 90/99 2026-02-22T22:23:13.235 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 91/99 2026-02-22T22:23:13.236 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 92/99 2026-02-22T22:23:13.236 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 93/99 2026-02-22T22:23:13.236 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 94/99 2026-02-22T22:23:13.236 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 95/99 2026-02-22T22:23:13.236 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 96/99 2026-02-22T22:23:13.236 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 97/99 2026-02-22T22:23:13.236 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 98/99 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 99/99 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-grafana-dashboards-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mds-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-k8sevents-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-modules-core-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-mon-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-osd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-prometheus-alerts-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: cephadm-2:19.2.3-689.gf90f4eae.el9.noarch 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-22T22:23:13.290 INFO:teuthology.orchestra.run.trial123.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: lua-5.4.4-4.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-22T22:23:13.291 INFO:teuthology.orchestra.run.trial123.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-logutils-0.3.5-21.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-mako-1.1.4-6.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-22T22:23:13.292 INFO:teuthology.orchestra.run.trial123.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-pecan-1.4.2-3.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-ply-3.11-14.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-22T22:23:13.293 INFO:teuthology.orchestra.run.trial123.stdout: python3-webob-1.8.8-2.el9.noarch 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: unzip-6.0-59.el9.x86_64 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: zip-3.0-35.el9.x86_64 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:13.294 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:13.345 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-base 2026-02-22T22:23:13.655 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:49 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:13.719 INFO:teuthology.orchestra.run.trial123.stdout:Package ceph-base-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:13.737 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:13.738 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:13.738 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:13.758 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install cephadm 2026-02-22T22:23:14.068 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:50 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:14.132 INFO:teuthology.orchestra.run.trial123.stdout:Package cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T22:23:14.150 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:14.151 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:14.151 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:14.172 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-immutable-object-cache 2026-02-22T22:23:14.481 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:50 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:14.565 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:14.565 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:14.565 INFO:teuthology.orchestra.run.trial123.stdout: Package Arch Version Repo Size 2026-02-22T22:23:14.565 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:14.565 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout: ceph-immutable-object-cache x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 145 k 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout:Install 1 Package 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 145 k 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 431 k 2026-02-22T22:23:14.566 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:23:14.653 INFO:teuthology.orchestra.run.trial123.stdout:ceph-immutable-object-cache-19.2.3-689.gf90f4ea 1.6 MB/s | 145 kB 00:00 2026-02-22T22:23:14.654 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:14.654 INFO:teuthology.orchestra.run.trial123.stdout:Total 1.6 MB/s | 145 kB 00:00 2026-02-22T22:23:14.654 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:14.658 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:14.658 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:14.689 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:14.690 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:14.744 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:14.746 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T22:23:14.768 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T22:23:14.768 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:23:14.768 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-22T22:23:14.768 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:15.062 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.e 1/1 2026-02-22T22:23:15.063 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:15.063 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:15.063 INFO:teuthology.orchestra.run.trial123.stdout: ceph-immutable-object-cache-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:15.063 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:15.063 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:15.089 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-mgr 2026-02-22T22:23:15.401 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:51 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:15.467 INFO:teuthology.orchestra.run.trial123.stdout:Package ceph-mgr-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:15.486 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:15.486 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:15.487 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:15.507 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-mgr-dashboard 2026-02-22T22:23:15.817 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:51 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:15.883 INFO:teuthology.orchestra.run.trial123.stdout:Package ceph-mgr-dashboard-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T22:23:15.901 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:15.902 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:15.902 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:15.924 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-22T22:23:16.230 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:52 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:16.295 INFO:teuthology.orchestra.run.trial123.stdout:Package ceph-mgr-diskprediction-local-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T22:23:16.314 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:16.314 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:16.314 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:16.335 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-mgr-rook 2026-02-22T22:23:16.640 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:52 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:16.708 INFO:teuthology.orchestra.run.trial123.stdout:Package ceph-mgr-rook-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T22:23:16.726 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:16.727 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:16.727 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:16.747 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-mgr-cephadm 2026-02-22T22:23:17.054 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:53 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:17.118 INFO:teuthology.orchestra.run.trial123.stdout:Package ceph-mgr-cephadm-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T22:23:17.136 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:17.137 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:17.137 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:17.159 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-fuse 2026-02-22T22:23:17.468 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:53 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:17.552 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:17.552 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:17.552 INFO:teuthology.orchestra.run.trial123.stdout: Package Architecture Version Repository Size 2026-02-22T22:23:17.552 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:17.552 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:23:17.552 INFO:teuthology.orchestra.run.trial123.stdout: ceph-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 1.2 M 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout:Install 1 Package 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 1.2 M 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 3.6 M 2026-02-22T22:23:17.553 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:23:17.869 INFO:teuthology.orchestra.run.trial123.stdout:ceph-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 3.6 MB/s | 1.2 MB 00:00 2026-02-22T22:23:17.869 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:17.869 INFO:teuthology.orchestra.run.trial123.stdout:Total 3.6 MB/s | 1.2 MB 00:00 2026-02-22T22:23:17.869 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:17.873 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:17.873 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:17.910 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:17.910 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:18.013 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:18.030 INFO:teuthology.orchestra.run.trial123.stdout: Installing : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:18.278 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:18.334 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:18.334 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:18.334 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:18.335 INFO:teuthology.orchestra.run.trial123.stdout: ceph-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:18.335 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:18.335 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:18.362 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install ceph-volume 2026-02-22T22:23:18.675 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:54 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:18.739 INFO:teuthology.orchestra.run.trial123.stdout:Package ceph-volume-2:19.2.3-689.gf90f4eae.el9.noarch is already installed. 2026-02-22T22:23:18.758 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:18.758 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:18.759 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:18.780 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install librados-devel 2026-02-22T22:23:19.085 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:55 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout: Package Arch Version Repo Size 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout: librados-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 126 k 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:23:19.169 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:19.170 INFO:teuthology.orchestra.run.trial123.stdout:Install 1 Package 2026-02-22T22:23:19.170 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:19.170 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 126 k 2026-02-22T22:23:19.170 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 456 k 2026-02-22T22:23:19.170 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:23:19.254 INFO:teuthology.orchestra.run.trial123.stdout:librados-devel-19.2.3-689.gf90f4eae.el9.x86_64. 1.5 MB/s | 126 kB 00:00 2026-02-22T22:23:19.255 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:19.255 INFO:teuthology.orchestra.run.trial123.stdout:Total 1.4 MB/s | 126 kB 00:00 2026-02-22T22:23:19.255 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:19.257 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:19.257 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:19.289 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:19.290 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:19.344 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:19.360 INFO:teuthology.orchestra.run.trial123.stdout: Installing : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:19.444 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:19.501 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:19.502 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:19.502 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:19.502 INFO:teuthology.orchestra.run.trial123.stdout: librados-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:19.502 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:19.502 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:19.531 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install libcephfs2 2026-02-22T22:23:19.849 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:55 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:19.914 INFO:teuthology.orchestra.run.trial123.stdout:Package libcephfs2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:19.932 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:19.933 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:19.933 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:19.954 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install libcephfs-devel 2026-02-22T22:23:20.268 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:56 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:20.352 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout: Package Arch Version Repo Size 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout: libcephfs-devel x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 34 k 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:23:20.353 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:20.354 INFO:teuthology.orchestra.run.trial123.stdout:Install 1 Package 2026-02-22T22:23:20.354 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:20.354 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 34 k 2026-02-22T22:23:20.354 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 153 k 2026-02-22T22:23:20.354 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:23:20.635 INFO:teuthology.orchestra.run.trial123.stdout:libcephfs-devel-19.2.3-689.gf90f4eae.el9.x86_64 119 kB/s | 34 kB 00:00 2026-02-22T22:23:20.635 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:20.636 INFO:teuthology.orchestra.run.trial123.stdout:Total 119 kB/s | 34 kB 00:00 2026-02-22T22:23:20.636 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:20.638 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:20.638 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:20.641 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:20.641 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:20.661 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:20.677 INFO:teuthology.orchestra.run.trial123.stdout: Installing : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:20.734 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:20.791 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:20.791 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:20.791 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:20.792 INFO:teuthology.orchestra.run.trial123.stdout: libcephfs-devel-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:20.792 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:20.792 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:20.821 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install librados2 2026-02-22T22:23:21.131 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:57 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:21.197 INFO:teuthology.orchestra.run.trial123.stdout:Package librados2-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:21.216 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:21.216 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:21.216 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:21.238 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install librbd1 2026-02-22T22:23:21.552 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:57 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:21.617 INFO:teuthology.orchestra.run.trial123.stdout:Package librbd1-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:21.636 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:21.636 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:21.636 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:21.658 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install python3-rados 2026-02-22T22:23:21.969 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:57 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:22.036 INFO:teuthology.orchestra.run.trial123.stdout:Package python3-rados-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:22.054 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:22.055 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:22.055 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:22.076 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install python3-rgw 2026-02-22T22:23:22.389 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:58 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:22.455 INFO:teuthology.orchestra.run.trial123.stdout:Package python3-rgw-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:22.473 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:22.474 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:22.474 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:22.495 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install python3-cephfs 2026-02-22T22:23:22.804 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:58 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:22.870 INFO:teuthology.orchestra.run.trial123.stdout:Package python3-cephfs-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:22.888 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:22.889 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:22.889 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:22.911 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install python3-rbd 2026-02-22T22:23:23.222 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:59 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:23.287 INFO:teuthology.orchestra.run.trial123.stdout:Package python3-rbd-2:19.2.3-689.gf90f4eae.el9.x86_64 is already installed. 2026-02-22T22:23:23.305 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:23.306 INFO:teuthology.orchestra.run.trial123.stdout:Nothing to do. 2026-02-22T22:23:23.306 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:23.327 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install rbd-fuse 2026-02-22T22:23:23.633 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:00:59 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:23.717 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:23.717 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:23.717 INFO:teuthology.orchestra.run.trial123.stdout: Package Architecture Version Repository Size 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout: rbd-fuse x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 85 k 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout:Install 1 Package 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:23.718 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 85 k 2026-02-22T22:23:23.719 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 227 k 2026-02-22T22:23:23.719 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:23:23.839 INFO:teuthology.orchestra.run.trial123.stdout:rbd-fuse-19.2.3-689.gf90f4eae.el9.x86_64.rpm 697 kB/s | 85 kB 00:00 2026-02-22T22:23:23.840 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:23.840 INFO:teuthology.orchestra.run.trial123.stdout:Total 692 kB/s | 85 kB 00:00 2026-02-22T22:23:23.841 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:23.843 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:23.843 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:23.874 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:23.874 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:23.925 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:23.942 INFO:teuthology.orchestra.run.trial123.stdout: Installing : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:24.021 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:24.078 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:24.079 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:24.079 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:24.079 INFO:teuthology.orchestra.run.trial123.stdout: rbd-fuse-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:24.079 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:24.079 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:24.105 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install rbd-mirror 2026-02-22T22:23:24.418 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:01:00 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout: Package Arch Version Repo Size 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout: rbd-mirror x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 3.1 M 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:24.504 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:23:24.505 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:24.505 INFO:teuthology.orchestra.run.trial123.stdout:Install 1 Package 2026-02-22T22:23:24.505 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:24.505 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 3.1 M 2026-02-22T22:23:24.505 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 13 M 2026-02-22T22:23:24.505 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:23:24.817 INFO:teuthology.orchestra.run.trial123.stdout:rbd-mirror-19.2.3-689.gf90f4eae.el9.x86_64.rpm 10 MB/s | 3.1 MB 00:00 2026-02-22T22:23:24.818 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:24.818 INFO:teuthology.orchestra.run.trial123.stdout:Total 9.9 MB/s | 3.1 MB 00:00 2026-02-22T22:23:24.818 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:24.821 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:24.821 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:24.862 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:24.862 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:25.058 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:25.060 INFO:teuthology.orchestra.run.trial123.stdout: Installing : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:25.083 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:25.084 INFO:teuthology.orchestra.run.trial123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-22T22:23:25.084 INFO:teuthology.orchestra.run.trial123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-22T22:23:25.084 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T22:23:25.084 INFO:teuthology.orchestra.run.trial123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-22T22:23:25.084 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:25.382 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:25.382 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:25.382 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:25.383 INFO:teuthology.orchestra.run.trial123.stdout: rbd-mirror-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:25.383 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:25.383 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:25.412 DEBUG:teuthology.orchestra.run.trial123:> sudo yum -y install rbd-nbd 2026-02-22T22:23:25.718 INFO:teuthology.orchestra.run.trial123.stdout:Last metadata expiration check: 0:01:01 ago on Sun 22 Feb 2026 10:22:24 PM UTC. 2026-02-22T22:23:25.802 INFO:teuthology.orchestra.run.trial123.stdout:Dependencies resolved. 2026-02-22T22:23:25.802 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:25.802 INFO:teuthology.orchestra.run.trial123.stdout: Package Architecture Version Repository Size 2026-02-22T22:23:25.802 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:25.802 INFO:teuthology.orchestra.run.trial123.stdout:Installing: 2026-02-22T22:23:25.802 INFO:teuthology.orchestra.run.trial123.stdout: rbd-nbd x86_64 2:19.2.3-689.gf90f4eae.el9 ceph 170 k 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout:Transaction Summary 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout:================================================================================ 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout:Install 1 Package 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout:Total download size: 170 k 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout:Installed size: 490 k 2026-02-22T22:23:25.803 INFO:teuthology.orchestra.run.trial123.stdout:Downloading Packages: 2026-02-22T22:23:25.890 INFO:teuthology.orchestra.run.trial123.stdout:rbd-nbd-19.2.3-689.gf90f4eae.el9.x86_64.rpm 1.9 MB/s | 170 kB 00:00 2026-02-22T22:23:25.890 INFO:teuthology.orchestra.run.trial123.stdout:-------------------------------------------------------------------------------- 2026-02-22T22:23:25.891 INFO:teuthology.orchestra.run.trial123.stdout:Total 1.9 MB/s | 170 kB 00:00 2026-02-22T22:23:25.891 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction check 2026-02-22T22:23:25.894 INFO:teuthology.orchestra.run.trial123.stdout:Transaction check succeeded. 2026-02-22T22:23:25.894 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction test 2026-02-22T22:23:25.925 INFO:teuthology.orchestra.run.trial123.stdout:Transaction test succeeded. 2026-02-22T22:23:25.925 INFO:teuthology.orchestra.run.trial123.stdout:Running transaction 2026-02-22T22:23:25.983 INFO:teuthology.orchestra.run.trial123.stdout: Preparing : 1/1 2026-02-22T22:23:25.999 INFO:teuthology.orchestra.run.trial123.stdout: Installing : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:26.081 INFO:teuthology.orchestra.run.trial123.stdout: Running scriptlet: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:26.140 INFO:teuthology.orchestra.run.trial123.stdout: Verifying : rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 1/1 2026-02-22T22:23:26.140 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:26.140 INFO:teuthology.orchestra.run.trial123.stdout:Installed: 2026-02-22T22:23:26.140 INFO:teuthology.orchestra.run.trial123.stdout: rbd-nbd-2:19.2.3-689.gf90f4eae.el9.x86_64 2026-02-22T22:23:26.141 INFO:teuthology.orchestra.run.trial123.stdout: 2026-02-22T22:23:26.141 INFO:teuthology.orchestra.run.trial123.stdout:Complete! 2026-02-22T22:23:26.172 DEBUG:teuthology.parallel:result is None 2026-02-22T22:23:26.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:23:26.339 DEBUG:teuthology.orchestra.run.trial123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-22T22:23:26.360 INFO:teuthology.orchestra.run.trial123.stdout:19.2.3-689.gf90f4eae.el9 2026-02-22T22:23:26.360 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689.gf90f4eae.el9 2026-02-22T22:23:26.360 INFO:teuthology.task.install:The correct ceph version 19.2.3-689.gf90f4eae is installed. 2026-02-22T22:23:26.363 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T22:23:26.363 DEBUG:teuthology.orchestra.run.trial123:> set -ex 2026-02-22T22:23:26.363 DEBUG:teuthology.orchestra.run.trial123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T22:23:26.425 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T22:23:26.425 DEBUG:teuthology.orchestra.run.trial123:> set -ex 2026-02-22T22:23:26.425 DEBUG:teuthology.orchestra.run.trial123:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T22:23:26.489 DEBUG:teuthology.orchestra.run.trial123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T22:23:26.555 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T22:23:26.555 DEBUG:teuthology.orchestra.run.trial123:> set -ex 2026-02-22T22:23:26.555 DEBUG:teuthology.orchestra.run.trial123:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T22:23:26.618 DEBUG:teuthology.orchestra.run.trial123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T22:23:26.682 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T22:23:26.682 DEBUG:teuthology.orchestra.run.trial123:> set -ex 2026-02-22T22:23:26.682 DEBUG:teuthology.orchestra.run.trial123:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T22:23:26.746 DEBUG:teuthology.orchestra.run.trial123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T22:23:26.809 INFO:teuthology.run_tasks:Running task workunit... 2026-02-22T22:23:26.834 INFO:tasks.workunit:Pulling workunits from ref f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:23:26.835 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-22T22:23:26.835 INFO:tasks.workunit:timeout=3h 2026-02-22T22:23:26.835 INFO:tasks.workunit:cleanup=True 2026-02-22T22:23:26.835 DEBUG:teuthology.orchestra.run.trial123:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-22T22:23:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:23:26.866 INFO:teuthology.orchestra.run.trial123.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-22T22:23:26.866 DEBUG:teuthology.orchestra.run.trial123:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-22T22:23:26.922 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-22T22:23:26.922 DEBUG:teuthology.orchestra.run.trial123:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-22T22:23:26.979 DEBUG:teuthology.orchestra.run.trial123:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T22:23:27.041 INFO:tasks.workunit.client.0.trial123.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-22T22:23:46.522 INFO:tasks.workunit.client.0.trial123.stderr:Note: switching to 'f90f4eaebb28b619cf76e80cbbe997656da69579'. 2026-02-22T22:23:46.522 INFO:tasks.workunit.client.0.trial123.stderr: 2026-02-22T22:23:46.522 INFO:tasks.workunit.client.0.trial123.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-22T22:23:46.522 INFO:tasks.workunit.client.0.trial123.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr:state without impacting any branches by switching back to a branch. 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr: 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr: 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr: git switch -c 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr: 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr:Or undo this operation with: 2026-02-22T22:23:46.523 INFO:tasks.workunit.client.0.trial123.stderr: 2026-02-22T22:23:46.524 INFO:tasks.workunit.client.0.trial123.stderr: git switch - 2026-02-22T22:23:46.524 INFO:tasks.workunit.client.0.trial123.stderr: 2026-02-22T22:23:46.524 INFO:tasks.workunit.client.0.trial123.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-22T22:23:46.524 INFO:tasks.workunit.client.0.trial123.stderr: 2026-02-22T22:23:46.524 INFO:tasks.workunit.client.0.trial123.stderr:HEAD is now at f90f4eaebb2 Merge branch 'wip-74712-squid' of https://github.com/idryomov/ceph into wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T22:23:46.530 DEBUG:teuthology.orchestra.run.trial123:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-22T22:23:46.590 DEBUG:teuthology.orchestra.run.trial123:> set -ex 2026-02-22T22:23:46.591 DEBUG:teuthology.orchestra.run.trial123:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-22T22:23:46.649 INFO:tasks.workunit:Running workunits matching misc on client.0... 2026-02-22T22:23:46.650 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2026-02-22T22:23:46.651 DEBUG:teuthology.orchestra.run.trial123:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2026-02-22T22:23:46.712 INFO:tasks.workunit.client.0.trial123.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T22:23:46.716 INFO:tasks.workunit.client.0.trial123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T22:23:46.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/mclock-config 2026-02-22T22:23:46.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2026-02-22T22:23:46.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2026-02-22T22:23:46.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2026-02-22T22:23:46.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2026-02-22T22:23:46.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2026-02-22T22:23:46.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2026-02-22T22:23:46.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none ' 2026-02-22T22:23:46.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-22T22:23:46.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2026-02-22T22:23:46.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2026-02-22T22:23:46.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:TEST_profile_builtin_to_custom 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:TEST_profile_custom_to_builtin 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:TEST_profile_disallow_builtin_params_modify 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:TEST_profile_disallow_builtin_params_override 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:TEST_recovery_limit_adjustment_mclock' 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:23:46.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:23:46.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:23:46.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:23:46.722 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:23:46.722 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:23:46.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:23:46.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:23:46.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:23:46.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:23:46.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:23:46.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:23:46.724 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:23:46.724 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:23:46.724 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:23:46.724 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:23:46.725 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:23:46.725 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:23:46.725 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:23:46.725 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:23:46.726 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:23:46.726 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:23:46.726 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:23:46.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:23:46.728 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.728 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:23:46.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:23:46.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:23:46.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:23:46.729 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:23:46.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:23:46.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:23:46.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T22:23:46.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:23:46.733 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:23:46.734 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:23:46.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:23:46.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:23:46.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:23:46.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:23:46.735 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:23:46.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:23:46.735 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:23:46.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:23:46.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:23:46.736 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:23:46.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:23:46.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:23:46.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:23:46.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:23:46.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:23:46.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:23:46.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:23:46.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:23:46.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:23:46.740 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:23:46.740 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:23:46.740 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:23:46.741 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:23:46.741 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.742 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.742 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:23:46.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:23:46.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:23:46.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2026-02-22T22:23:46.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-22T22:23:46.762 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:23:46.762 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:23:46.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:23:46.763 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:23:46.763 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.763 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:23:46.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:23:46.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:23:46.789 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:23:46.790 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.790 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.790 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:23:46.790 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:23:46.790 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get fsid 2026-02-22T22:23:46.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:23:46.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.835 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:23:46.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:23:46.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get mon_host 2026-02-22T22:23:46.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-22T22:23:46.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2026-02-22T22:23:46.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:23:46.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:23:46.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:23:46.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2026-02-22T22:23:46.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:23:46.975 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:23:46.975 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:23:46.975 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:23:46.975 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:23:46.975 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.975 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.975 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:23:46.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:23:46.976 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-22T22:23:46.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:23:46.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-22T22:23:46.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2026-02-22T22:23:46.996 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:23:46.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=834ce8f5-f6d1-422e-b624-995a08a55905 2026-02-22T22:23:46.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 834ce8f5-f6d1-422e-b624-995a08a55905' 2026-02-22T22:23:46.996 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 834ce8f5-f6d1-422e-b624-995a08a55905 2026-02-22T22:23:46.997 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:23:47.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzgZtpNBFDABAAoI6xFzQiHVTGHfc5N/K8WA== 2026-02-22T22:23:47.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzgZtpNBFDABAAoI6xFzQiHVTGHfc5N/K8WA=="}' 2026-02-22T22:23:47.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 834ce8f5-f6d1-422e-b624-995a08a55905 -i td/mclock-config/0/new.json 2026-02-22T22:23:47.106 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:23:47.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2026-02-22T22:23:47.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDzgZtpNBFDABAAoI6xFzQiHVTGHfc5N/K8WA== --osd-uuid 834ce8f5-f6d1-422e-b624-995a08a55905 2026-02-22T22:23:47.131 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:47.128+0000 7f8021227780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:23:47.132 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:47.130+0000 7f8021227780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:23:47.133 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:47.131+0000 7f8021227780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:23:47.133 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:47.131+0000 7f8021227780 -1 bdev(0x5625a946bc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:23:47.134 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:47.131+0000 7f8021227780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-22T22:23:49.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-22T22:23:49.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:23:49.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:23:49.544 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:23:49.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:23:49.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:23:49.798 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:23:49.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-22T22:23:49.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:23:49.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:23:49.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:23:49.813 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:49.811+0000 7f2727818780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:23:49.815 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:49.813+0000 7f2727818780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:23:49.816 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:49.814+0000 7f2727818780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:23:49.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:23:49.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:23:49.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:23:49.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:23:49.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:23:49.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:23:49.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:23:49.986 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:23:49.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:23:49.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:23:50.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:23:51.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:23:51.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:23:51.172 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:23:51.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:23:51.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:23:51.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:23:51.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:23:51.398 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:51.395+0000 7f2727818780 -1 Falling back to public interface 2026-02-22T22:23:52.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:23:52.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:23:52.362 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:23:52.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:23:52.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:23:52.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:23:52.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:23:53.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:23:53.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:23:53.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:23:53.548 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:23:53.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:23:53.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:23:53.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:23:54.362 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:23:54.360+0000 7f2727818780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:23:54.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:23:54.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:23:54.738 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:23:54.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:23:54.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:23:54.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:23:54.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:23:55.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:23:55.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:23:55.929 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:23:55.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:23:55.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:23:55.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:23:56.114 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3019236046,v1:127.0.0.1:6803/3019236046] [v2:127.0.0.1:6804/3019236046,v1:127.0.0.1:6805/3019236046] exists,up 834ce8f5-f6d1-422e-b624-995a08a55905 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:23:56.116 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:23:56.117 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:23:56.117 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:56.117 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:56.117 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:23:56.117 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:23:56.117 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:23:56.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2026-02-22T22:23:56.162 INFO:tasks.workunit.client.0.trial123.stdout:osd_max_backfills: 1 2026-02-22T22:23:56.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2026-02-22T22:23:56.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-22T22:23:56.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-22T22:23:58.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-22T22:23:58.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:23:58.356 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:23:58.356 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:23:58.356 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:23:58.357 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:23:58.357 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:58.357 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:58.357 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:23:58.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:23:58.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:23:58.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2026-02-22T22:23:58.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-22T22:23:58.405 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-22T22:23:58.405 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:23:58.405 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:23:58.406 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok dump_recovery_reservations 2026-02-22T22:23:58.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-22T22:23:58.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-22T22:23:58.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-22T22:23:58.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:23:58.456 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:23:58.456 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:23:58.456 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:23:58.457 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:23:58.457 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:23:58.457 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:23:58.457 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:23:58.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:23:58.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok dump_recovery_reservations 2026-02-22T22:23:58.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-22T22:23:58.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-22T22:23:58.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-22T22:23:58.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-22T22:23:58.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-22T22:24:00.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-22T22:24:00.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:24:00.881 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:24:00.881 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:00.881 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:00.881 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:00.881 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:00.882 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:00.882 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:00.882 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:24:00.882 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:24:00.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-22T22:24:00.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-22T22:24:00.931 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:24:00.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok dump_recovery_reservations 2026-02-22T22:24:00.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-22T22:24:00.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-22T22:24:00.979 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-22T22:24:00.979 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:24:00.979 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:24:00.980 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok dump_recovery_reservations 2026-02-22T22:24:01.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-22T22:24:01.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-22T22:24:01.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2026-02-22T22:24:01.025 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:01.025 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:01.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:01.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:01.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:02.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:02.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2026-02-22T22:24:02.520 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 is already down. 2026-02-22T22:24:02.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2026-02-22T22:24:02.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2026-02-22T22:24:02.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:24:02.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:24:02.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:24:02.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:02.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:24:02.528 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:02.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:02.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2026-02-22T22:24:02.711 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/3019236046,v1:127.0.0.1:6803/3019236046] [v2:127.0.0.1:6804/3019236046,v1:127.0.0.1:6805/3019236046] exists 834ce8f5-f6d1-422e-b624-995a08a55905 2026-02-22T22:24:02.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:24:02.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:24:02.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:24:02.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2026-02-22T22:24:02.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:24:02.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T22:24:02.715 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:24:02.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2026-02-22T22:24:02.716 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2026-02-22T22:24:02.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T22:24:02.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:24:02.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:24:02.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:24:02.730 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:02.727+0000 7f5e0ac25780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:02.731 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:02.728+0000 7f5e0ac25780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:02.732 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:02.729+0000 7f5e0ac25780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:02.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T22:24:02.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:24:02.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:24:02.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:24:02.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:24:02.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:02.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:24:02.902 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:02.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:02.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:03.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:04.062 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:04.059+0000 7f5e0ac25780 -1 Falling back to public interface 2026-02-22T22:24:04.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:04.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:04.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:24:04.086 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:24:04.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:04.087 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:04.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:05.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:05.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:05.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:24:05.276 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:24:05.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:05.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:05.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:06.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:06.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:06.465 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:24:06.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:24:06.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:06.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:06.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:06.759 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:06.756+0000 7f5e0ac25780 -1 osd.0 6 log_to_monitors true 2026-02-22T22:24:07.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:07.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:07.653 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:24:07.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:24:07.654 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:07.654 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:07.840 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 6 last_clean_interval [5,6) [v2:127.0.0.1:6802/2718167271,v1:127.0.0.1:6803/2718167271] [v2:127.0.0.1:6804/2718167271,v1:127.0.0.1:6805/2718167271] exists,up 834ce8f5-f6d1-422e-b624-995a08a55905 2026-02-22T22:24:07.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:24:07.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:24:07.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:24:07.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-22T22:24:07.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:24:07.842 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:24:07.842 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:07.843 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:07.843 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:07.843 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:07.843 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:07.843 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:07.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:24:07.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:24:07.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-22T22:24:07.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-22T22:24:07.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-22T22:24:07.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:24:07.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:24:07.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:07.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:07.891 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:07.891 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:07.892 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:07.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:07.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:24:07.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok dump_recovery_reservations 2026-02-22T22:24:07.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-22T22:24:07.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-22T22:24:07.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:24:07.939 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok dump_recovery_reservations 2026-02-22T22:24:07.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-22T22:24:07.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-22T22:24:07.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2026-02-22T22:24:07.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:07.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:07.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:07.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:07.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:07.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:07.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:07.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:08.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:08.093 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:08.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:08.094 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:08.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:08.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:08.096 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:08.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:08.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:08.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:08.097 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:08.098 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:08.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:08.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:08.103 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:08.103 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.103 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.103 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:08.105 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:08.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:08.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:08.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:08.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:08.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:08.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:08.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:08.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:08.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:08.109 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:08.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:08.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:08.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:08.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:08.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:08.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:08.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:08.113 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:08.113 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.113 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:08.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:08.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:08.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:08.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:08.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:08.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:08.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:08.119 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:08.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:08.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:08.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:08.120 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:08.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:08.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:08.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:08.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:08.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:08.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:08.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:08.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:24:08.125 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:24:08.125 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.125 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:24:08.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:24:08.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:24:08.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:24:08.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2026-02-22T22:24:08.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:08.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:24:08.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:24:08.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:24:08.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:08.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:08.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:24:08.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:24:08.172 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:08.173 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:08.173 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:08.173 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:08.173 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.173 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.173 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:08.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:08.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get fsid 2026-02-22T22:24:08.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:24:08.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:08.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:08.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:24:08.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:24:08.218 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:08.218 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:08.218 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:08.218 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:08.218 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.218 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.219 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:08.219 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:08.219 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get mon_host 2026-02-22T22:24:08.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2026-02-22T22:24:08.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2026-02-22T22:24:08.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:24:08.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:24:08.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:24:08.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2026-02-22T22:24:08.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:24:08.360 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:24:08.360 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:08.360 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:08.361 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:08.361 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.361 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:08.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:08.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:24:08.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:08.382 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:24:08.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:24:08.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-22T22:24:08.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2026-02-22T22:24:08.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:24:08.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b9c2849-c586-4886-9b29-22011c03194d 2026-02-22T22:24:08.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4b9c2849-c586-4886-9b29-22011c03194d' 2026-02-22T22:24:08.384 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 4b9c2849-c586-4886-9b29-22011c03194d 2026-02-22T22:24:08.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:24:08.391 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIgptpXtgwFxAAj+4m/p3IELfIMPzUHghUdA== 2026-02-22T22:24:08.391 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIgptpXtgwFxAAj+4m/p3IELfIMPzUHghUdA=="}' 2026-02-22T22:24:08.392 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b9c2849-c586-4886-9b29-22011c03194d -i td/mclock-config/0/new.json 2026-02-22T22:24:08.493 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:08.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2026-02-22T22:24:08.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAIgptpXtgwFxAAj+4m/p3IELfIMPzUHghUdA== --osd-uuid 4b9c2849-c586-4886-9b29-22011c03194d 2026-02-22T22:24:08.519 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:08.517+0000 7fb4a8e20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:08.521 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:08.519+0000 7fb4a8e20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:08.522 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:08.519+0000 7fb4a8e20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:08.523 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:08.520+0000 7fb4a8e20780 -1 bdev(0x5640d4df3c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:24:08.523 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:08.520+0000 7fb4a8e20780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-22T22:24:10.923 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-22T22:24:10.923 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:24:10.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:24:10.924 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:24:10.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:24:11.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:24:11.176 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:24:11.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-22T22:24:11.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:24:11.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:24:11.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:24:11.192 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:11.189+0000 7fc2ed40f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:11.193 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:11.191+0000 7fc2ed40f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:11.194 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:11.192+0000 7fc2ed40f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:11.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:24:11.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:24:11.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:24:11.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:24:11.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:24:11.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:11.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:24:11.364 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:11.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:11.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:11.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:12.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:12.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:12.550 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:24:12.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:24:12.551 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:12.551 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:12.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:13.035 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:13.033+0000 7fc2ed40f780 -1 Falling back to public interface 2026-02-22T22:24:13.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:13.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:13.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:24:13.737 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:24:13.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:13.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:13.923 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:14.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:14.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:14.924 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:24:14.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:24:14.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:14.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:15.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:15.726 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:15.723+0000 7fc2ed40f780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:24:16.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:16.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:16.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:24:16.112 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:24:16.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:16.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:16.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:17.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:17.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:17.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:24:17.306 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:24:17.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:17.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:17.492 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/185172622,v1:127.0.0.1:6803/185172622] [v2:127.0.0.1:6804/185172622,v1:127.0.0.1:6805/185172622] exists,up 4b9c2849-c586-4886-9b29-22011c03194d 2026-02-22T22:24:17.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:24:17.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:24:17.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:24:17.493 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2026-02-22T22:24:17.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2026-02-22T22:24:17.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-22T22:24:17.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_profile 2026-02-22T22:24:17.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2026-02-22T22:24:17.728 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2026-02-22T22:24:17.729 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2026-02-22T22:24:17.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2026-02-22T22:24:17.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2026-02-22T22:24:17.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-22T22:24:17.784 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:24:17.784 INFO:tasks.workunit.client.0.trial123.stdout: "success": "" 2026-02-22T22:24:17.784 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:24:17.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-22T22:24:17.789 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-22T22:24:17.789 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:17.790 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:17.790 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:17.790 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:17.790 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:17.790 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:17.790 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-22T22:24:17.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_profile 2026-02-22T22:24:17.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2026-02-22T22:24:17.838 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2026-02-22T22:24:17.838 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2026-02-22T22:24:17.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2026-02-22T22:24:17.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2026-02-22T22:24:17.840 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:17.840 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2026-02-22T22:24:17.840 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-22T22:24:17.840 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:17.840 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:17.840 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:17.841 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:17.841 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:17.841 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:17.841 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-22T22:24:17.841 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:17.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2026-02-22T22:24:17.889 INFO:tasks.workunit.client.0.trial123.stdout:client_res = 0.300000 2026-02-22T22:24:17.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2026-02-22T22:24:17.890 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2026-02-22T22:24:17.890 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2026-02-22T22:24:17.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2026-02-22T22:24:17.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2026-02-22T22:24:17.891 INFO:tasks.workunit.client.0.trial123.stdout:client_res_new = .400000 2026-02-22T22:24:17.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2026-02-22T22:24:18.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-22T22:24:18.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2026-02-22T22:24:18.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2026-02-22T22:24:18.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2026-02-22T22:24:18.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.268 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:18.269 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-22T22:24:18.269 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:18.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2026-02-22T22:24:18.315 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2026-02-22T22:24:18.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2026-02-22T22:24:18.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-22T22:24:18.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2026-02-22T22:24:18.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:18.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:18.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:18.317 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:18.317 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:18.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:18.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:18.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:18.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:18.423 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:18.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:18.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:18.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:18.426 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:18.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:18.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:18.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:18.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.428 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:18.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:18.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:18.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:18.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:18.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:18.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:18.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:18.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:18.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:18.437 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:18.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:18.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:18.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:18.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:18.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:18.440 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:18.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:18.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:18.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:18.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:18.445 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:18.446 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:18.446 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:18.446 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:18.446 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:18.446 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:18.446 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:18.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:18.447 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:18.448 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:18.448 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:18.449 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:18.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:18.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:18.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:18.451 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:18.451 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:18.452 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:18.453 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.453 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.453 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:18.453 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:18.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:18.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:24:18.455 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:24:18.455 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.455 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:24:18.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:18.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:18.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:18.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:18.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:18.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:18.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:18.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:18.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:18.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:18.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:18.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:18.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:18.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:18.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:18.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:18.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:18.462 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:18.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:18.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:18.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:18.465 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:18.465 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.465 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:18.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:18.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:18.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:24:18.467 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:24:18.467 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.467 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:24:18.468 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:24:18.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:24:18.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2026-02-22T22:24:18.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-22T22:24:18.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:24:18.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:18.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:18.488 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:18.488 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.488 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:18.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:24:18.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:18.515 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:18.516 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.516 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.516 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:18.516 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:18.516 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get fsid 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:18.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:18.562 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:18.563 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.563 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.563 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:18.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:18.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get mon_host 2026-02-22T22:24:18.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2026-02-22T22:24:18.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2026-02-22T22:24:18.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:24:18.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:24:18.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:24:18.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2026-02-22T22:24:18.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:24:18.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:24:18.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:18.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:18.708 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:18.708 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.708 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:18.709 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:18.709 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:18.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-22T22:24:18.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:24:18.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:24:18.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa3e11d5-997c-416e-8196-0cccb7e93c41 2026-02-22T22:24:18.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aa3e11d5-997c-416e-8196-0cccb7e93c41' 2026-02-22T22:24:18.732 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 aa3e11d5-997c-416e-8196-0cccb7e93c41 2026-02-22T22:24:18.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:24:18.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASgptp/srsKxAAgXPbRNw45VJWzfuiE4n15g== 2026-02-22T22:24:18.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASgptp/srsKxAAgXPbRNw45VJWzfuiE4n15g=="}' 2026-02-22T22:24:18.740 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa3e11d5-997c-416e-8196-0cccb7e93c41 -i td/mclock-config/0/new.json 2026-02-22T22:24:18.841 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:18.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2026-02-22T22:24:18.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQASgptp/srsKxAAgXPbRNw45VJWzfuiE4n15g== --osd-uuid aa3e11d5-997c-416e-8196-0cccb7e93c41 2026-02-22T22:24:18.866 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:18.863+0000 7f48a830e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:18.867 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:18.865+0000 7f48a830e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:18.868 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:18.866+0000 7f48a830e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:18.869 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:18.866+0000 7f48a830e780 -1 bdev(0x55efaee97c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:24:18.869 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:18.866+0000 7f48a830e780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-22T22:24:21.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-22T22:24:21.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:24:21.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:24:21.263 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:24:21.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:24:21.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:24:21.518 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:24:21.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-22T22:24:21.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:24:21.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:24:21.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:24:21.535 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:21.533+0000 7f0d17027780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:21.536 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:21.534+0000 7f0d17027780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:21.537 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:21.535+0000 7f0d17027780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:21.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:21.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:22.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:22.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:22.894 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:24:22.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:24:22.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:22.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:23.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:23.123 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:23.120+0000 7f0d17027780 -1 Falling back to public interface 2026-02-22T22:24:24.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:24.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:24.082 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:24:24.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:24:24.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:24.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:24.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:25.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:25.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:25.269 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:24:25.269 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:24:25.269 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:25.269 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:25.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:26.073 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:26.069+0000 7f0d17027780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:24:26.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:26.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:26.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:24:26.456 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:24:26.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:26.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:26.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:27.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:27.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:27.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:24:27.653 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:24:27.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:27.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:27.837 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3321116355,v1:127.0.0.1:6803/3321116355] [v2:127.0.0.1:6804/3321116355,v1:127.0.0.1:6805/3321116355] exists,up aa3e11d5-997c-416e-8196-0cccb7e93c41 2026-02-22T22:24:27.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:24:27.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:24:27.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:24:27.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2026-02-22T22:24:27.840 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2026-02-22T22:24:28.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2026-02-22T22:24:28.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2026-02-22T22:24:28.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-22T22:24:28.026 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-22T22:24:28.026 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:28.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:28.027 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:28.027 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:28.027 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:28.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:28.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-22T22:24:28.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_profile 2026-02-22T22:24:28.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2026-02-22T22:24:28.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2026-02-22T22:24:28.074 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2026-02-22T22:24:28.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2026-02-22T22:24:28.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-22T22:24:28.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-22T22:24:28.129 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:24:28.129 INFO:tasks.workunit.client.0.trial123.stdout: "success": "" 2026-02-22T22:24:28.129 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:24:28.134 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-22T22:24:28.134 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-22T22:24:28.134 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:28.135 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:28.135 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:28.135 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:28.135 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:28.135 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:28.135 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-22T22:24:28.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_profile 2026-02-22T22:24:28.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2026-02-22T22:24:28.186 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2026-02-22T22:24:28.186 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2026-02-22T22:24:28.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2026-02-22T22:24:28.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2026-02-22T22:24:28.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2026-02-22T22:24:28.187 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:28.187 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2026-02-22T22:24:28.187 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-22T22:24:28.187 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:28.187 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:28.187 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:28.188 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:28.188 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:28.188 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:28.188 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-22T22:24:28.188 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:28.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2026-02-22T22:24:28.233 INFO:tasks.workunit.client.0.trial123.stdout:Original client_res for osd.0 = 0.300000 2026-02-22T22:24:28.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2026-02-22T22:24:28.234 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2026-02-22T22:24:28.234 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2026-02-22T22:24:28.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2026-02-22T22:24:28.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2026-02-22T22:24:28.235 INFO:tasks.workunit.client.0.trial123.stdout:client_res_new = .400000 2026-02-22T22:24:28.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2026-02-22T22:24:28.421 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-22T22:24:28.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2026-02-22T22:24:28.610 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-22T22:24:28.610 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2026-02-22T22:24:28.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:28.612 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:28.613 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:28.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-22T22:24:28.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:28.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2026-02-22T22:24:28.660 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-22T22:24:28.660 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2026-02-22T22:24:28.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-22T22:24:28.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2026-02-22T22:24:28.716 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:24:28.716 INFO:tasks.workunit.client.0.trial123.stdout: "success": "" 2026-02-22T22:24:28.716 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:24:28.721 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-22T22:24:28.721 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-22T22:24:28.721 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:28.722 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:28.722 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:28.722 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:28.722 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:28.722 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:28.722 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-22T22:24:28.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_profile 2026-02-22T22:24:28.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2026-02-22T22:24:28.769 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2026-02-22T22:24:28.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-22T22:24:28.769 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-22T22:24:28.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2026-02-22T22:24:28.954 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-22T22:24:28.954 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2026-02-22T22:24:28.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-22T22:24:28.956 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:28.956 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2026-02-22T22:24:28.956 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-22T22:24:28.956 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:28.956 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:28.957 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:28.957 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:28.957 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:28.957 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:28.957 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-22T22:24:28.957 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:29.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2026-02-22T22:24:29.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-22T22:24:29.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2026-02-22T22:24:29.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-22T22:24:29.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2026-02-22T22:24:29.192 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2026-02-22T22:24:34.193 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-22T22:24:34.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2026-02-22T22:24:34.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2026-02-22T22:24:34.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2026-02-22T22:24:34.385 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-22T22:24:34.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:34.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2026-02-22T22:24:34.386 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-22T22:24:34.386 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:34.387 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:34.387 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:34.388 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.388 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.388 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:34.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-22T22:24:34.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:34.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2026-02-22T22:24:34.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2026-02-22T22:24:34.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2026-02-22T22:24:34.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-22T22:24:34.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2026-02-22T22:24:34.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:34.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:34.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:34.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:34.437 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:34.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:34.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:34.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:34.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:34.543 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:34.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:34.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:34.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:34.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:34.546 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:34.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:34.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:34.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:34.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:34.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:34.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:34.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:34.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:34.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:34.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-22T22:24:34.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:34.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:34.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:34.555 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:34.555 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:34.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:34.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:34.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:34.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:34.557 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:34.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:34.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:34.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:34.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:34.559 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:34.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:34.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:34.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:34.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:34.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:34.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:34.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:34.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:34.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:34.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-22T22:24:34.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:34.567 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:34.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:34.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:34.568 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:34.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:34.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:34.569 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:34.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:34.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:34.571 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:34.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:34.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:34.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:34.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:34.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:34.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:34.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:24:34.576 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:24:34.576 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.576 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.576 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:24:34.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:24:34.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:24:34.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:24:34.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:24:34.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2026-02-22T22:24:34.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2026-02-22T22:24:34.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2026-02-22T22:24:34.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:34.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:34.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:34.581 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:34.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:34.581 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:34.582 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:34.582 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:34.582 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:34.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:34.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:34.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:34.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:34.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:34.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:34.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:34.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.587 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:34.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:34.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:34.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:24:34.588 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:24:34.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:24:34.590 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:24:34.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:24:34.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:24:34.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:24:34.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2026-02-22T22:24:34.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2026-02-22T22:24:34.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:24:34.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:24:34.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:24:34.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2026-02-22T22:24:34.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-22T22:24:34.610 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:24:34.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:34.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:34.611 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:34.611 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.611 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:34.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:24:34.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:24:34.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:24:34.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:34.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:34.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:24:34.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:34.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get fsid 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:34.685 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:34.686 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:34.686 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.686 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.686 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:34.686 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:34.686 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get mon_host 2026-02-22T22:24:34.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2026-02-22T22:24:34.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2026-02-22T22:24:34.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:24:34.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:24:34.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:24:34.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2026-02-22T22:24:34.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:24:34.827 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:24:34.827 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:34.827 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:34.827 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:34.828 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.828 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.828 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:34.828 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:34.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:34.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-22T22:24:34.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:34.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2026-02-22T22:24:34.850 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:24:34.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e5f202c4-f46e-4a4d-ac5b-60600c3ceba9 2026-02-22T22:24:34.851 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 e5f202c4-f46e-4a4d-ac5b-60600c3ceba9 2026-02-22T22:24:34.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e5f202c4-f46e-4a4d-ac5b-60600c3ceba9' 2026-02-22T22:24:34.851 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:24:34.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAigptp7j8WMxAAFG8lMDAAGIZSs/7xazWGTw== 2026-02-22T22:24:34.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAigptp7j8WMxAAFG8lMDAAGIZSs/7xazWGTw=="}' 2026-02-22T22:24:34.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e5f202c4-f46e-4a4d-ac5b-60600c3ceba9 -i td/mclock-config/0/new.json 2026-02-22T22:24:34.962 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:34.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2026-02-22T22:24:34.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAigptp7j8WMxAAFG8lMDAAGIZSs/7xazWGTw== --osd-uuid e5f202c4-f46e-4a4d-ac5b-60600c3ceba9 2026-02-22T22:24:34.986 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:34.985+0000 7fa088e14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:34.988 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:34.986+0000 7fa088e14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:34.989 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:34.987+0000 7fa088e14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:34.989 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:34.987+0000 7fa088e14780 -1 bdev(0x55f81c1f1c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:24:34.989 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:34.987+0000 7fa088e14780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-22T22:24:37.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-22T22:24:37.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:24:37.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:24:37.374 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:24:37.375 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:24:37.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:24:37.629 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:24:37.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-22T22:24:37.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:24:37.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:24:37.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:24:37.643 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:37.641+0000 7f830ee16780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:37.644 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:37.643+0000 7f830ee16780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:37.645 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:37.644+0000 7f830ee16780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:37.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:24:37.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:24:37.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:24:37.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:24:37.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:24:37.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:37.815 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:37.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:24:37.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:37.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:38.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:38.974 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:38.971+0000 7f830ee16780 -1 Falling back to public interface 2026-02-22T22:24:39.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:39.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:39.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:24:39.004 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:24:39.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:39.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:39.190 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:40.191 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:40.191 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:40.192 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:24:40.192 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:24:40.192 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:40.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:40.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:41.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:41.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:41.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:24:41.377 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:24:41.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:41.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:41.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:42.148 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:42.147+0000 7f830ee16780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:24:42.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:42.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:42.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:24:42.565 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:24:42.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:42.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:42.751 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1513462054,v1:127.0.0.1:6803/1513462054] [v2:127.0.0.1:6804/1513462054,v1:127.0.0.1:6805/1513462054] exists,up e5f202c4-f46e-4a4d-ac5b-60600c3ceba9 2026-02-22T22:24:42.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:24:42.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:24:42.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:24:42.752 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2026-02-22T22:24:42.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2026-02-22T22:24:42.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2026-02-22T22:24:42.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2026-02-22T22:24:42.941 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-22T22:24:42.941 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:42.941 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:42.941 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:42.941 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:42.941 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:42.942 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:42.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-22T22:24:42.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_profile 2026-02-22T22:24:42.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2026-02-22T22:24:42.989 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2026-02-22T22:24:42.989 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2026-02-22T22:24:42.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2026-02-22T22:24:42.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2026-02-22T22:24:42.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2026-02-22T22:24:42.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2026-02-22T22:24:42.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2026-02-22T22:24:42.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2026-02-22T22:24:42.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2026-02-22T22:24:42.991 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:42.991 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2026-02-22T22:24:42.991 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-22T22:24:42.991 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:42.991 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:42.991 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:42.991 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:42.992 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:42.992 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:42.992 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-22T22:24:42.992 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:43.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2026-02-22T22:24:43.036 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2026-02-22T22:24:43.036 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:43.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2026-02-22T22:24:43.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2026-02-22T22:24:43.231 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2026-02-22T22:24:43.231 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2026-02-22T22:24:43.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2026-02-22T22:24:43.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2026-02-22T22:24:43.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2026-02-22T22:24:45.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2026-02-22T22:24:45.234 INFO:tasks.workunit.client.0.trial123.stdout:Check configuration values - Attempt#: 0 2026-02-22T22:24:45.234 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:45.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2026-02-22T22:24:45.424 INFO:tasks.workunit.client.0.trial123.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2026-02-22T22:24:45.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2026-02-22T22:24:45.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2026-02-22T22:24:45.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:45.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2026-02-22T22:24:45.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2026-02-22T22:24:45.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:45.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2026-02-22T22:24:45.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2026-02-22T22:24:45.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2026-02-22T22:24:45.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2026-02-22T22:24:45.619 INFO:tasks.workunit.client.0.trial123.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2026-02-22T22:24:45.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2026-02-22T22:24:45.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:45.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2026-02-22T22:24:45.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:45.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-22T22:24:45.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:45.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2026-02-22T22:24:45.624 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-22T22:24:45.624 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:45.624 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:45.625 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:45.625 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:45.625 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:45.625 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:45.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-22T22:24:45.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:45.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2026-02-22T22:24:45.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2026-02-22T22:24:45.669 INFO:tasks.workunit.client.0.trial123.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2026-02-22T22:24:45.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2026-02-22T22:24:45.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:45.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2026-02-22T22:24:45.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:45.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-22T22:24:45.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2026-02-22T22:24:45.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2026-02-22T22:24:45.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2026-02-22T22:24:45.673 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:45.673 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-22T22:24:45.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:45.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2026-02-22T22:24:45.721 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2026-02-22T22:24:45.721 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:45.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2026-02-22T22:24:45.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2026-02-22T22:24:45.912 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2026-02-22T22:24:45.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2026-02-22T22:24:45.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2026-02-22T22:24:45.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2026-02-22T22:24:45.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2026-02-22T22:24:47.914 INFO:tasks.workunit.client.0.trial123.stdout:Check configuration values - Attempt#: 0 2026-02-22T22:24:47.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2026-02-22T22:24:47.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-22T22:24:48.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2026-02-22T22:24:48.106 INFO:tasks.workunit.client.0.trial123.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2026-02-22T22:24:48.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2026-02-22T22:24:48.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2026-02-22T22:24:48.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:48.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2026-02-22T22:24:48.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2026-02-22T22:24:48.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2026-02-22T22:24:48.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2026-02-22T22:24:48.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2026-02-22T22:24:48.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2026-02-22T22:24:48.298 INFO:tasks.workunit.client.0.trial123.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2026-02-22T22:24:48.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2026-02-22T22:24:48.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2026-02-22T22:24:48.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:48.300 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2026-02-22T22:24:48.300 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:48.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-22T22:24:48.303 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:48.303 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2026-02-22T22:24:48.303 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-22T22:24:48.303 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:48.303 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:48.304 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:48.304 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.304 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.304 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:48.304 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-22T22:24:48.304 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:48.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2026-02-22T22:24:48.350 INFO:tasks.workunit.client.0.trial123.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2026-02-22T22:24:48.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2026-02-22T22:24:48.351 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2026-02-22T22:24:48.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:48.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2026-02-22T22:24:48.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-22T22:24:48.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-22T22:24:48.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2026-02-22T22:24:48.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2026-02-22T22:24:48.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:48.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:48.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:48.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:48.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:48.462 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:48.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:48.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:48.464 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:48.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:48.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:48.466 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:48.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:48.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:48.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:48.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:48.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:48.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:48.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:48.475 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:48.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:48.476 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:48.477 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:48.477 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:48.477 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:48.478 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.478 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:48.478 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:48.479 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.479 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:48.480 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:48.480 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:48.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:48.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:48.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:48.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:48.485 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:48.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:48.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:48.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:48.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:48.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:48.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:48.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:48.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:48.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:48.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:48.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:48.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:48.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:48.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:48.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:48.491 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:48.491 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.491 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.491 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:48.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:48.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:48.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:24:48.493 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:24:48.493 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.493 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:24:48.494 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:24:48.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:24:48.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:24:48.497 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:24:48.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:24:48.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:24:48.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:24:48.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:24:48.499 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:24:48.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:24:48.499 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:24:48.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:24:48.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:24:48.500 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:24:48.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:24:48.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:24:48.502 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:24:48.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:24:48.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:24:48.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:24:48.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:24:48.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:24:48.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:24:48.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:24:48.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:24:48.506 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:24:48.506 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.506 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:24:48.507 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:24:48.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:24:48.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2026-02-22T22:24:48.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:48.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:24:48.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:24:48.555 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:24:48.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:48.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:48.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:24:48.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:24:48.556 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:48.556 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:48.556 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:48.558 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:48.558 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.558 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.558 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:48.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:48.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get fsid 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:24:48.602 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:48.603 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.603 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.603 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:24:48.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:24:48.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get mon_host 2026-02-22T22:24:48.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2026-02-22T22:24:48.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2026-02-22T22:24:48.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:24:48.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:24:48.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:24:48.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2026-02-22T22:24:48.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:24:48.744 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:48.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:24:48.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:24:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:48.767 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:24:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:24:48.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-22T22:24:48.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2026-02-22T22:24:48.769 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:24:48.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7820268-3a3b-4c77-ae2a-e8aee34e9131 2026-02-22T22:24:48.769 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 f7820268-3a3b-4c77-ae2a-e8aee34e9131 2026-02-22T22:24:48.770 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f7820268-3a3b-4c77-ae2a-e8aee34e9131' 2026-02-22T22:24:48.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:24:48.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwgptpVSIxLhAAKNVmES3Q9fPyffgnufih8A== 2026-02-22T22:24:48.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwgptpVSIxLhAAKNVmES3Q9fPyffgnufih8A=="}' 2026-02-22T22:24:48.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7820268-3a3b-4c77-ae2a-e8aee34e9131 -i td/mclock-config/0/new.json 2026-02-22T22:24:48.880 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:48.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2026-02-22T22:24:48.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAwgptpVSIxLhAAKNVmES3Q9fPyffgnufih8A== --osd-uuid f7820268-3a3b-4c77-ae2a-e8aee34e9131 2026-02-22T22:24:48.906 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:48.903+0000 7f90c6a27780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:48.907 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:48.905+0000 7f90c6a27780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:48.908 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:48.905+0000 7f90c6a27780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:48.909 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:48.906+0000 7f90c6a27780 -1 bdev(0x55a3cc3f1c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:24:48.909 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:48.906+0000 7f90c6a27780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-22T22:24:51.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-22T22:24:51.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:24:51.312 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:24:51.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:24:51.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:24:51.568 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:24:51.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:24:51.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-22T22:24:51.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:24:51.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:24:51.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:24:51.585 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:51.582+0000 7f18b120d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:51.586 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:51.583+0000 7f18b120d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:51.587 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:51.584+0000 7f18b120d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:24:51.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:24:51.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:24:51.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:24:51.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:24:51.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:24:51.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:51.757 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:24:51.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:24:51.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:51.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:51.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:52.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:52.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:52.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:24:52.943 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:24:52.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:52.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:53.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:53.170 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:53.167+0000 7f18b120d780 -1 Falling back to public interface 2026-02-22T22:24:54.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:54.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:54.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:24:54.130 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:24:54.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:54.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:54.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:55.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:55.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:55.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:24:55.317 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:24:55.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:55.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:55.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:24:55.851 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:24:55.849+0000 7f18b120d780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:24:56.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:24:56.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:24:56.504 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:24:56.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:24:56.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:24:56.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:24:56.694 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3711181987,v1:127.0.0.1:6803/3711181987] [v2:127.0.0.1:6804/3711181987,v1:127.0.0.1:6805/3711181987] exists,up f7820268-3a3b-4c77-ae2a-e8aee34e9131 2026-02-22T22:24:56.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:24:56.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:24:56.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:24:56.695 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2026-02-22T22:24:56.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2026-02-22T22:24:56.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2026-02-22T22:24:56.884 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2026-02-22T22:24:56.884 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-22T22:24:56.884 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:56.885 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:56.885 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:56.885 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:56.885 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:56.885 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:56.885 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-22T22:24:56.885 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_profile 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2026-02-22T22:24:56.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2026-02-22T22:24:56.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:56.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2026-02-22T22:24:56.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-22T22:24:56.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:56.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:56.933 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:56.933 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:56.933 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:56.933 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:56.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-22T22:24:56.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:56.983 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2026-02-22T22:24:56.983 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2026-02-22T22:24:56.983 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:24:56.984 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2026-02-22T22:24:56.984 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2026-02-22T22:24:57.041 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:24:57.042 INFO:tasks.workunit.client.0.trial123.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2026-02-22T22:24:57.042 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:24:57.046 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2026-02-22T22:24:57.047 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2026-02-22T22:24:57.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2026-02-22T22:24:57.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2026-02-22T22:24:57.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2026-02-22T22:24:59.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2026-02-22T22:24:59.048 INFO:tasks.workunit.client.0.trial123.stdout:Check configuration values - Attempt#: 0 2026-02-22T22:24:59.049 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:59.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2026-02-22T22:24:59.238 INFO:tasks.workunit.client.0.trial123.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2026-02-22T22:24:59.238 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2026-02-22T22:24:59.238 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2026-02-22T22:24:59.238 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:24:59.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2026-02-22T22:24:59.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2026-02-22T22:24:59.241 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:59.241 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2026-02-22T22:24:59.241 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2026-02-22T22:24:59.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2026-02-22T22:24:59.430 INFO:tasks.workunit.client.0.trial123.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2026-02-22T22:24:59.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2026-02-22T22:24:59.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2026-02-22T22:24:59.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:24:59.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2026-02-22T22:24:59.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:24:59.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-22T22:24:59.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:59.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2026-02-22T22:24:59.435 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-22T22:24:59.435 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:59.435 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:59.435 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:59.435 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:59.436 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:59.436 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:59.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-22T22:24:59.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-22T22:24:59.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2026-02-22T22:24:59.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2026-02-22T22:24:59.483 INFO:tasks.workunit.client.0.trial123.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2026-02-22T22:24:59.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2026-02-22T22:24:59.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:24:59.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2026-02-22T22:24:59.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:24:59.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-22T22:24:59.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2026-02-22T22:24:59.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2026-02-22T22:24:59.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2026-02-22T22:24:59.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2026-02-22T22:24:59.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2026-02-22T22:24:59.487 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-22T22:24:59.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:24:59.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2026-02-22T22:24:59.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2026-02-22T22:24:59.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:24:59.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2026-02-22T22:24:59.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2026-02-22T22:24:59.588 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:24:59.588 INFO:tasks.workunit.client.0.trial123.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2026-02-22T22:24:59.588 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:24:59.593 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2026-02-22T22:24:59.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2026-02-22T22:24:59.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2026-02-22T22:24:59.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2026-02-22T22:24:59.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2026-02-22T22:25:01.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2026-02-22T22:25:01.595 INFO:tasks.workunit.client.0.trial123.stdout:Check configuration values - Attempt#: 0 2026-02-22T22:25:01.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-22T22:25:01.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2026-02-22T22:25:01.783 INFO:tasks.workunit.client.0.trial123.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2026-02-22T22:25:01.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2026-02-22T22:25:01.784 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2026-02-22T22:25:01.784 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:25:01.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2026-02-22T22:25:01.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2026-02-22T22:25:01.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2026-02-22T22:25:01.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2026-02-22T22:25:01.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2026-02-22T22:25:01.970 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2026-02-22T22:25:01.971 INFO:tasks.workunit.client.0.trial123.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2026-02-22T22:25:01.971 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2026-02-22T22:25:01.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2026-02-22T22:25:01.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:25:01.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2026-02-22T22:25:01.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:25:01.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:01.975 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:01.976 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:25:01.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-22T22:25:01.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-22T22:25:02.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2026-02-22T22:25:02.022 INFO:tasks.workunit.client.0.trial123.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2026-02-22T22:25:02.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2026-02-22T22:25:02.023 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2026-02-22T22:25:02.023 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:25:02.024 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2026-02-22T22:25:02.024 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-22T22:25:02.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-22T22:25:02.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2026-02-22T22:25:02.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2026-02-22T22:25:02.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2026-02-22T22:25:02.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:25:02.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:25:02.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:25:02.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:02.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:02.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:02.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:02.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:02.133 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:02.133 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:02.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:02.134 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:02.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:02.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:02.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:02.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:02.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:02.138 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:02.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:25:02.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:25:02.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:02.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:25:02.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:02.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:02.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-22T22:25:02.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:25:02.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:25:02.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:25:02.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:02.145 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:02.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:02.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:02.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:02.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:02.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:02.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:02.147 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:02.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:02.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:02.148 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:02.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:02.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:02.150 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:02.151 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:25:02.151 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:25:02.152 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:02.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:02.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:02.156 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:02.156 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:02.157 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:02.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:02.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:02.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:02.158 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:02.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:02.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:02.159 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:02.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:25:02.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:25:02.162 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:02.162 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.162 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:25:02.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:02.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:02.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:25:02.164 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:25:02.164 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.164 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2026-02-22T22:25:02.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:25:02.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:25:02.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:25:02.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:02.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:02.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:02.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:02.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:02.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:02.169 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:02.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:02.170 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:02.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:02.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:02.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:02.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:02.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:02.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:02.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:02.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:25:02.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:25:02.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:02.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:25:02.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:02.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:02.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2026-02-22T22:25:02.177 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:25:02.177 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.177 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37404 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2026-02-22T22:25:02.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-22T22:25:02.197 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:25:02.198 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:02.198 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:02.198 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:02.198 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.198 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.198 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:25:02.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:25:02.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:25:02.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:25:02.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:25:02.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:25:02.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:25:02.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:25:02.225 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:25:02.225 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:25:02.225 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:25:02.226 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:02.226 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.226 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.226 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:25:02.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:25:02.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get fsid 2026-02-22T22:25:02.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:25:02.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:25:02.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.272 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-mon.a.asok 2026-02-22T22:25:02.273 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:25:02.273 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37404/ceph-mon.a.asok config get mon_host 2026-02-22T22:25:02.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-22T22:25:02.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2026-02-22T22:25:02.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:25:02.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:25:02.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:25:02.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2026-02-22T22:25:02.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:25:02.414 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:25:02.414 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:02.414 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:02.414 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:02.414 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.414 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.414 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:25:02.415 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:25:02.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:25:02.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:25:02.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' 2026-02-22T22:25:02.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:25:02.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=95e8b8ff-f349-4d2d-b332-127b06787fba 2026-02-22T22:25:02.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 95e8b8ff-f349-4d2d-b332-127b06787fba' 2026-02-22T22:25:02.437 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 95e8b8ff-f349-4d2d-b332-127b06787fba 2026-02-22T22:25:02.437 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:25:02.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+gptpJzRhGhAArVtWChIWeIYDOnMPyi1tfQ== 2026-02-22T22:25:02.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+gptpJzRhGhAArVtWChIWeIYDOnMPyi1tfQ=="}' 2026-02-22T22:25:02.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 95e8b8ff-f349-4d2d-b332-127b06787fba -i td/mclock-config/0/new.json 2026-02-22T22:25:02.545 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:25:02.552 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2026-02-22T22:25:02.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQA+gptpJzRhGhAArVtWChIWeIYDOnMPyi1tfQ== --osd-uuid 95e8b8ff-f349-4d2d-b332-127b06787fba 2026-02-22T22:25:02.571 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:02.569+0000 7f0b53415780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:02.572 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:02.570+0000 7f0b53415780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:02.573 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:02.571+0000 7f0b53415780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:02.573 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:02.571+0000 7f0b53415780 -1 bdev(0x55f206073c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:25:02.574 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:02.571+0000 7f0b53415780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-22T22:25:05.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-22T22:25:05.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:25:05.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:25:05.474 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:25:05.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:25:05.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:25:05.740 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:25:05.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7ad4dafd-d4ce-4537-8521-59895b136d99 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37404/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-22T22:25:05.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:25:05.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:25:05.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:25:05.755 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:05.753+0000 7f2d433a7780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:05.757 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:05.755+0000 7f2d433a7780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:05.758 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:05.756+0000 7f2d433a7780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:05.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:25:05.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:25:05.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:25:05.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:25:05.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:25:05.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:05.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:25:05.929 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:25:05.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:05.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:06.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:07.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:07.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:07.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:25:07.117 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:25:07.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:07.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:07.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:07.342 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:07.339+0000 7f2d433a7780 -1 Falling back to public interface 2026-02-22T22:25:08.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:08.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:08.309 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:25:08.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:25:08.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:08.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:08.495 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:09.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:09.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:09.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:25:09.497 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:25:09.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:09.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:09.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:10.295 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:10.293+0000 7f2d433a7780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:25:10.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:10.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:10.686 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:25:10.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:25:10.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:10.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:10.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:11.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:11.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:11.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:25:11.877 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:25:11.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:11.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:12.066 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1127077213,v1:127.0.0.1:6803/1127077213] [v2:127.0.0.1:6804/1127077213,v1:127.0.0.1:6805/1127077213] exists,up 95e8b8ff-f349-4d2d-b332-127b06787fba 2026-02-22T22:25:12.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:25:12.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:25:12.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:25:12.068 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:25:12.068 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:25:12.068 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:25:12.068 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:12.069 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:12.069 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:12.069 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:25:12.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:25:12.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-22T22:25:12.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2026-02-22T22:25:12.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2026-02-22T22:25:12.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2026-02-22T22:25:12.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-22T22:25:12.305 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-22T22:25:14.308 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:25:14.308 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:25:14.308 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:25:14.308 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:14.308 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:14.308 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:14.309 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:25:14.309 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:25:14.309 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-22T22:25:14.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2026-02-22T22:25:14.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2026-02-22T22:25:14.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-22T22:25:14.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-22T22:25:14.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-22T22:25:16.733 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-22T22:25:16.733 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:25:16.733 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:25:16.734 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:16.734 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:16.734 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:16.734 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37404/ceph-osd.0.asok 2026-02-22T22:25:16.734 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-22T22:25:16.735 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37404/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:16.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:16.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:16.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:16.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:16.890 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:16.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:16.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:16.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:16.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:16.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:16.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:16.894 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:16.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:16.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:25:16.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:25:16.899 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:16.899 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:16.899 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:16.899 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:25:16.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:16.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:16.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-22T22:25:16.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:25:16.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:25:16.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:25:16.901 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:16.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:16.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:16.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:16.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:16.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:16.903 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:16.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:16.904 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:16.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:16.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:16.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:16.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:16.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:16.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:16.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:16.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:16.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:25:16.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:25:16.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:16.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:16.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:16.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:25:16.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:16.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:16.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T22:25:16.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/mclock-config 0 2026-02-22T22:25:16.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2026-02-22T22:25:16.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T22:25:16.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2026-02-22T22:25:16.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:16.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:16.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:16.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:16.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:16.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:16.913 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:16.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:16.914 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:16.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:16.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:16.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:16.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:16.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:16.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:16.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:16.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:16.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T22:25:16.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2026-02-22T22:25:16.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:16.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:16.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37404 2026-02-22T22:25:16.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37404 2026-02-22T22:25:16.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:16.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:16.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T22:25:16.921 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T22:25:16.921 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T22:25:16.947 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2026-02-22T22:25:16.947 DEBUG:teuthology.orchestra.run.trial123:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2026-02-22T22:25:17.011 INFO:tasks.workunit.client.0.trial123.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T22:25:17.015 INFO:tasks.workunit.client.0.trial123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T22:25:17.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:25:17.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:25:17.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T22:25:17.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T22:25:17.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T22:25:17.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/network-ping 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2026-02-22T22:25:17.016 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2026-02-22T22:25:17.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none ' 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:TEST_network_ping_test2' 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-22T22:25:17.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:25:17.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:25:17.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:25:17.021 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:25:17.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:25:17.022 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:25:17.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:25:17.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:25:17.022 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:25:17.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:17.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:25:17.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:25:17.023 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:25:17.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:25:17.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:25:17.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2026-02-22T22:25:17.025 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:25:17.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:17.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:17.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.51626 2026-02-22T22:25:17.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:25:17.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:25:17.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2026-02-22T22:25:17.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:25:17.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:17.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:17.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.51626 2026-02-22T22:25:17.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:25:17.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T22:25:17.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T22:25:17.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:25:17.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-22T22:25:17.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2026-02-22T22:25:17.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2026-02-22T22:25:17.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2026-02-22T22:25:17.031 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2026-02-22T22:25:17.031 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:25:17.031 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:25:17.031 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:25:17.031 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2026-02-22T22:25:17.031 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-22T22:25:17.049 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:25:17.049 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:17.049 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:17.049 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:17.050 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:17.050 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:17.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:17.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:25:17.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:25:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:25:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:25:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:25:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:17.075 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mon.a.asok 2026-02-22T22:25:17.076 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:25:17.076 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.51626/ceph-mon.a.asok config get fsid 2026-02-22T22:25:17.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:25:17.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:25:17.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:25:17.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:25:17.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mon.a.asok 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:25:17.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.51626/ceph-mon.a.asok config get mon_host 2026-02-22T22:25:17.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2026-02-22T22:25:17.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2026-02-22T22:25:17.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:25:17.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:25:17.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:25:17.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2026-02-22T22:25:17.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:25:17.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:25:17.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:17.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:17.265 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:17.266 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:17.266 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:17.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:17.267 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:25:17.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:25:17.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2026-02-22T22:25:17.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-22T22:25:17.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:25:17.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:25:17.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:25:17.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2026-02-22T22:25:17.290 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:25:17.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e27bb2d4-431f-418c-8861-a0ff892ba782 2026-02-22T22:25:17.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e27bb2d4-431f-418c-8861-a0ff892ba782' 2026-02-22T22:25:17.291 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 e27bb2d4-431f-418c-8861-a0ff892ba782 2026-02-22T22:25:17.291 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:25:17.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNgptpGy+sERAAIawYBRYc3c84J6QDugbP1w== 2026-02-22T22:25:17.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNgptpGy+sERAAIawYBRYc3c84J6QDugbP1w=="}' 2026-02-22T22:25:17.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e27bb2d4-431f-418c-8861-a0ff892ba782 -i td/network-ping/0/new.json 2026-02-22T22:25:17.402 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:25:17.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2026-02-22T22:25:17.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNgptpGy+sERAAIawYBRYc3c84J6QDugbP1w== --osd-uuid e27bb2d4-431f-418c-8861-a0ff892ba782 2026-02-22T22:25:17.428 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:17.426+0000 7f2d3d519780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:17.429 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:17.427+0000 7f2d3d519780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:17.430 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:17.428+0000 7f2d3d519780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:17.430 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:17.428+0000 7f2d3d519780 -1 bdev(0x55b698f6fc00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:25:17.431 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:17.429+0000 7f2d3d519780 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-22T22:25:20.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-22T22:25:20.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:25:20.317 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:25:20.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:25:20.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:25:20.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:25:20.571 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:25:20.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:25:20.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:25:20.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:25:20.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:25:20.587 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:20.585+0000 7f6c0a19a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:20.588 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:20.586+0000 7f6c0a19a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:20.589 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:20.587+0000 7f6c0a19a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:20.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:25:20.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:25:20.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:25:20.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:25:20.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:25:20.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:20.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:25:20.758 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:25:20.759 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:20.759 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:20.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:21.942 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:21.939+0000 7f6c0a19a780 -1 Falling back to public interface 2026-02-22T22:25:21.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:21.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:21.948 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:25:21.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:25:21.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:21.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:22.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:23.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:23.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:23.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:25:23.136 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:25:23.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:23.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:23.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:24.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:24.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:24.324 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:25:24.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:25:24.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:24.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:24.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:24.872 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:24.869+0000 7f6c0a19a780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:25:25.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:25.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:25.510 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:25:25.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:25:25.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:25.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:25:25.694 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3148722601,v1:127.0.0.1:6803/3148722601] [v2:127.0.0.1:6804/3148722601,v1:127.0.0.1:6805/3148722601] exists,up e27bb2d4-431f-418c-8861-a0ff892ba782 2026-02-22T22:25:25.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:25:25.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:25.700 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:25:25.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:25:25.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2026-02-22T22:25:25.702 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:25:25.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e2d3c32-4a93-4e9c-ac68-5f75999340ed 2026-02-22T22:25:25.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9e2d3c32-4a93-4e9c-ac68-5f75999340ed' 2026-02-22T22:25:25.702 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 9e2d3c32-4a93-4e9c-ac68-5f75999340ed 2026-02-22T22:25:25.703 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:25:25.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVgptpNiI4KhAAG7/ZsVrzBihLC2lg+ziqoA== 2026-02-22T22:25:25.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVgptpNiI4KhAAG7/ZsVrzBihLC2lg+ziqoA=="}' 2026-02-22T22:25:25.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e2d3c32-4a93-4e9c-ac68-5f75999340ed -i td/network-ping/1/new.json 2026-02-22T22:25:25.901 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:25:25.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2026-02-22T22:25:25.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVgptpNiI4KhAAG7/ZsVrzBihLC2lg+ziqoA== --osd-uuid 9e2d3c32-4a93-4e9c-ac68-5f75999340ed 2026-02-22T22:25:25.928 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:25.925+0000 7f67e100f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:25.929 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:25.926+0000 7f67e100f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:25.930 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:25.927+0000 7f67e100f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:25.930 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:25.928+0000 7f67e100f780 -1 bdev(0x558aa127bc00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:25:25.930 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:25.928+0000 7f67e100f780 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-22T22:25:28.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-22T22:25:28.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:25:28.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:25:28.048 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:25:28.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:25:28.299 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:25:28.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:25:28.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:25:28.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:25:28.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:25:28.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:25:28.317 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:28.313+0000 7f3fedc10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:28.318 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:28.315+0000 7f3fedc10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:28.319 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:28.316+0000 7f3fedc10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:28.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:25:28.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:25:28.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:25:28.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:25:28.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:25:28.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:28.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:25:28.486 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:25:28.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:28.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:25:28.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:29.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:29.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:29.671 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:25:29.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:25:29.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:29.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:25:29.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:29.903 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:29.900+0000 7f3fedc10780 -1 Falling back to public interface 2026-02-22T22:25:30.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:30.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:30.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:25:30.862 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:25:30.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:30.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:25:31.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:32.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:32.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:32.050 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:25:32.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:25:32.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:32.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:25:32.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:32.837 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:32.835+0000 7f3fedc10780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:25:33.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:33.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:33.236 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:25:33.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:25:33.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:33.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:25:33.426 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:34.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:34.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:34.427 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:25:34.428 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:25:34.428 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:34.428 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:25:34.616 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/932964609,v1:127.0.0.1:6811/932964609] [v2:127.0.0.1:6812/932964609,v1:127.0.0.1:6813/932964609] exists,up 9e2d3c32-4a93-4e9c-ac68-5f75999340ed 2026-02-22T22:25:34.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2026-02-22T22:25:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:25:34.622 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:25:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:25:34.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:25:34.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:25:34.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2026-02-22T22:25:34.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:25:34.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c1b6ff63-b126-4471-bc29-39d006aad518 2026-02-22T22:25:34.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c1b6ff63-b126-4471-bc29-39d006aad518' 2026-02-22T22:25:34.624 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 c1b6ff63-b126-4471-bc29-39d006aad518 2026-02-22T22:25:34.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:25:34.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBegptpR5CeJRAAPbdQgkdWOH3haq8myNtyCg== 2026-02-22T22:25:34.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBegptpR5CeJRAAPbdQgkdWOH3haq8myNtyCg=="}' 2026-02-22T22:25:34.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c1b6ff63-b126-4471-bc29-39d006aad518 -i td/network-ping/2/new.json 2026-02-22T22:25:34.828 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:25:34.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2026-02-22T22:25:34.836 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBegptpR5CeJRAAPbdQgkdWOH3haq8myNtyCg== --osd-uuid c1b6ff63-b126-4471-bc29-39d006aad518 2026-02-22T22:25:34.855 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:34.853+0000 7f1d2a8fe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:34.856 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:34.854+0000 7f1d2a8fe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:34.857 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:34.855+0000 7f1d2a8fe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:34.857 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:34.856+0000 7f1d2a8fe780 -1 bdev(0x55db8cc17c00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:25:34.858 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:34.856+0000 7f1d2a8fe780 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-22T22:25:36.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-22T22:25:36.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:25:36.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:25:36.973 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:25:36.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:25:37.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:25:37.229 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:25:37.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:25:37.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:25:37.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:25:37.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:25:37.246 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:37.244+0000 7f8dd51d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:37.247 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:37.245+0000 7f8dd51d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:37.248 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:37.246+0000 7f8dd51d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:37.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:25:37.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:38.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:38.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:38.605 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:25:38.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:25:38.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:38.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:25:38.788 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:38.832 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:38.830+0000 7f8dd51d4780 -1 Falling back to public interface 2026-02-22T22:25:39.789 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:39.789 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:39.789 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:25:39.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:25:39.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:39.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:25:39.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:40.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:40.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:40.978 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:25:40.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:25:40.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:40.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:25:41.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:25:41.768 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:25:41.766+0000 7f8dd51d4780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:25:42.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:25:42.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:25:42.167 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:25:42.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:25:42.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:25:42.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:25:42.358 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2749200556,v1:127.0.0.1:6819/2749200556] [v2:127.0.0.1:6820/2749200556,v1:127.0.0.1:6821/2749200556] exists,up c1b6ff63-b126-4471-bc29-39d006aad518 2026-02-22T22:25:42.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:25:42.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:25:42.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:25:42.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2026-02-22T22:25:47.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2026-02-22T22:25:47.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2026-02-22T22:25:47.598 INFO:tasks.workunit.client.0.trial123.stderr:pool 'foo' already exists 2026-02-22T22:25:47.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:25:48.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-22T22:25:48.629 INFO:tasks.workunit.client.0.trial123.stdout:hints = 1 2026-02-22T22:25:48.630 INFO:tasks.workunit.client.0.trial123.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-22T22:25:48.630 INFO:tasks.workunit.client.0.trial123.stdout:Object prefix: benchmark_data_trial123_56314 2026-02-22T22:25:48.630 INFO:tasks.workunit.client.0.trial123.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T22:25:48.630 INFO:tasks.workunit.client.0.trial123.stdout: 0 0 0 0 0 0 - 0 2026-02-22T22:25:49.629 INFO:tasks.workunit.client.0.trial123.stdout: 1 16 10756 10740 41.9423 41.9531 0.00115666 0.00148795 2026-02-22T22:25:50.629 INFO:tasks.workunit.client.0.trial123.stdout: 2 16 22252 22236 43.4225 44.9062 0.00142843 0.00143761 2026-02-22T22:25:51.629 INFO:tasks.workunit.client.0.trial123.stdout: 3 16 34078 34062 44.3454 46.1953 0.0010243 0.00140768 2026-02-22T22:25:52.630 INFO:tasks.workunit.client.0.trial123.stdout: 4 16 46306 46290 45.1994 47.7656 0.00143666 0.00138134 2026-02-22T22:25:53.629 INFO:tasks.workunit.client.0.trial123.stdout: 5 16 57172 57156 44.6479 42.4453 0.00154026 0.00139843 2026-02-22T22:25:54.630 INFO:tasks.workunit.client.0.trial123.stdout: 6 16 66583 66567 43.3333 36.7617 0.00172324 0.00144086 2026-02-22T22:25:55.629 INFO:tasks.workunit.client.0.trial123.stdout: 7 16 75787 75771 42.2786 35.9531 0.00167501 0.00147689 2026-02-22T22:25:56.630 INFO:tasks.workunit.client.0.trial123.stdout: 8 16 85066 85050 41.5242 36.2461 0.00132248 0.00150373 2026-02-22T22:25:57.630 INFO:tasks.workunit.client.0.trial123.stdout: 9 16 94283 94267 40.9106 36.0039 0.00155067 0.00152627 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Total time run: 10.0008 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Total writes made: 103685 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Write size: 4096 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Object size: 4096 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Bandwidth (MB/sec): 40.4985 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Stddev Bandwidth: 4.76904 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Max bandwidth (MB/sec): 47.7656 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Min bandwidth (MB/sec): 35.9531 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Average IOPS: 10367 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Stddev IOPS: 1220.87 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Max IOPS: 12228 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Min IOPS: 9204 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Average Latency(s): 0.00154201 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Stddev Latency(s): 0.000638449 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Max latency(s): 0.0127218 2026-02-22T22:25:58.630 INFO:tasks.workunit.client.0.trial123.stdout:Min latency(s): 0.000543517 2026-02-22T22:25:58.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2026-02-22T22:26:08.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2026-02-22T22:26:08.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:26:08.634 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:26:08.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:26:08.825 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:26:08.825 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T22:26:08.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:26:08.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:26:08.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:26:08.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836491 2026-02-22T22:26:08.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836491 2026-02-22T22:26:08.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T22:26:08.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:26:08.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:26:08.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672969 2026-02-22T22:26:08.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672969 2026-02-22T22:26:08.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2026-02-22T22:26:08.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:26:08.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:26:09.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509448 2026-02-22T22:26:09.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509448 2026-02-22T22:26:09.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448' 2026-02-22T22:26:09.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:26:09.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836491 2026-02-22T22:26:09.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:26:09.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:26:09.012 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836491 2026-02-22T22:26:09.013 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:26:09.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836491 2026-02-22T22:26:09.017 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836491 2026-02-22T22:26:09.017 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T22:26:09.017 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:26:09.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-22T22:26:09.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:26:10.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:26:10.202 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:26:10.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T22:26:10.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:26:10.394 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672969 2026-02-22T22:26:10.394 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:26:10.395 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:26:10.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672969 2026-02-22T22:26:10.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:26:10.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672969 2026-02-22T22:26:10.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2026-02-22T22:26:10.397 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672969 2026-02-22T22:26:10.398 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:26:10.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672970 -lt 42949672969 2026-02-22T22:26:10.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:26:10.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509448 2026-02-22T22:26:10.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:26:10.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:26:10.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509448 2026-02-22T22:26:10.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:26:10.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509448 2026-02-22T22:26:10.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2026-02-22T22:26:10.587 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509448 2026-02-22T22:26:10.587 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:26:10.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509448 -lt 64424509448 2026-02-22T22:26:10.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:10.831 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-osd.0.asok 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:10.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-osd.0.asok dump_osd_network 2026-02-22T22:26:10.905 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:10.905 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 1000, 2026-02-22T22:26:10.905 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:26:10.905 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:10.910 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:10.910 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:10.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:10.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:10.913 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:10.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2026-02-22T22:26:10.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:10.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-22T22:26:10.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2026-02-22T22:26:10.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2026-02-22T22:26:10.916 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:10.916 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:10.916 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:10.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mgr.x.asok 2026-02-22T22:26:10.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:10.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-mgr.x.asok dump_osd_network 2026-02-22T22:26:10.956 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:10.956 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 1000, 2026-02-22T22:26:10.956 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:26:10.956 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:10.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:10.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:10.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:10.989 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:10.989 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:10.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2026-02-22T22:26:10.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:10.993 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2026-02-22T22:26:10.993 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:26:10.993 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:26:10.993 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:10.993 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:10.993 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:10.994 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-osd.0.asok 2026-02-22T22:26:10.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:10.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-osd.0.asok dump_osd_network 0 2026-02-22T22:26:11.037 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:11.037 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 0, 2026-02-22T22:26:11.037 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [ 2026-02-22T22:26:11.037 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.037 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.037 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.037 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.123, 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.145, 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.151 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.131, 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.184, 2026-02-22T22:26:11.038 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.184 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.131, 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.184, 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.184 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.141 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.039 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.114, 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.142, 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.140 2026-02-22T22:26:11.040 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.138, 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.374, 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.374 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.138, 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.374, 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.374 2026-02-22T22:26:11.041 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.129 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.114, 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.129, 2026-02-22T22:26:11.042 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.131 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.121, 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.231, 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.231 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.121, 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.231, 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.231 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.043 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.113 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.083, 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.113, 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.109 2026-02-22T22:26:11.044 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.091, 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.313, 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.313 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.091, 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.313, 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.313 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.045 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.096 2026-02-22T22:26:11.046 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:26:11.046 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:26:11.046 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:11.049 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:11.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:11.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2026-02-22T22:26:11.075 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:11.076 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:11.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:11.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:11.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mgr.x.asok 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:11.078 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-mgr.x.asok dump_osd_network 0 2026-02-22T22:26:11.119 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:11.119 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 0, 2026-02-22T22:26:11.119 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [ 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.127, 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.241, 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.219 2026-02-22T22:26:11.120 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.117, 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.117, 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.117 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.150, 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 2.029, 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 2.029 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.134 2026-02-22T22:26:11.121 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.117, 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.219, 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.170 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.122 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.087, 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.087, 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.087 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.180, 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 1.993, 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 1.993 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.116 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.123 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.115, 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.133, 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.175 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.124 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.098, 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.080, 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.080 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.135, 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.226, 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.286 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.135 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.125 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.134, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.166, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.171 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.102, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.102, 2026-02-22T22:26:11.126 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.102 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.166, 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.424, 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.424 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.147 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:11.127 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.143, 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.157, 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.161 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.121, 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.095, 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.095 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.128 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.175, 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.301, 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.301 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.175 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.129 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.098, 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.122, 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.161 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.090, 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.084, 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.084 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.117, 2026-02-22T22:26:11.130 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.321, 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.321 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.117 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.139, 2026-02-22T22:26:11.131 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.148, 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.155 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.093, 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.093, 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.093 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.194, 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.262, 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.262 2026-02-22T22:26:11.132 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.161 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.111, 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.149, 2026-02-22T22:26:11.133 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.151 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.077, 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.077, 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.077 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.149, 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.280, 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.280 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.149 2026-02-22T22:26:11.134 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.123, 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.145, 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.151 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.135 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.109, 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.101, 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.101 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.131, 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.184, 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.184 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.131 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.136 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.114, 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.142, 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.140 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.103, 2026-02-22T22:26:11.137 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.089, 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.089 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.138, 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.374, 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.374 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.110 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.138 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.114, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.129, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.131 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.098, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.078, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.078 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.139 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.121, 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.231, 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.231 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.118 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:08 2026", 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:11.140 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.083, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.113, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.109 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.075, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.067, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.067 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.091, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.313, 2026-02-22T22:26:11.141 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.313 2026-02-22T22:26:11.142 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:11.142 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.091 2026-02-22T22:26:11.142 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:26:11.142 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:26:11.142 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:11.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:11.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:11.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2026-02-22T22:26:11.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:11.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:11.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:11.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2026-02-22T22:26:31.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2026-02-22T22:26:31.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:26:31.130 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:26:31.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:26:31.323 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:26:31.323 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T22:26:31.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:26:31.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:26:31.323 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:26:31.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836498 2026-02-22T22:26:31.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836498 2026-02-22T22:26:31.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836498' 2026-02-22T22:26:31.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:26:31.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:26:31.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672976 2026-02-22T22:26:31.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672976 2026-02-22T22:26:31.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2026-02-22T22:26:31.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:26:31.447 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:26:31.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509454 2026-02-22T22:26:31.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509454 2026-02-22T22:26:31.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454' 2026-02-22T22:26:31.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:26:31.510 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836498 2026-02-22T22:26:31.510 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:26:31.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:26:31.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836498 2026-02-22T22:26:31.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:26:31.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836498 2026-02-22T22:26:31.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2026-02-22T22:26:31.513 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836498 2026-02-22T22:26:31.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:26:31.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836496 -lt 21474836498 2026-02-22T22:26:31.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:26:32.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:26:32.701 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:26:32.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836496 -lt 21474836498 2026-02-22T22:26:32.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:26:33.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:26:33.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:26:34.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836498 -lt 21474836498 2026-02-22T22:26:34.076 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:26:34.076 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672976 2026-02-22T22:26:34.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:26:34.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:26:34.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672976 2026-02-22T22:26:34.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:26:34.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672976 2026-02-22T22:26:34.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2026-02-22T22:26:34.079 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672976 2026-02-22T22:26:34.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:26:34.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672976 -lt 42949672976 2026-02-22T22:26:34.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:26:34.269 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509454 2026-02-22T22:26:34.269 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:26:34.270 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:26:34.270 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509454 2026-02-22T22:26:34.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:26:34.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509454 2026-02-22T22:26:34.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2026-02-22T22:26:34.272 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509454 2026-02-22T22:26:34.273 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:26:34.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509455 -lt 64424509454 2026-02-22T22:26:34.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:34.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2026-02-22T22:26:34.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:26:34.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:26:34.462 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:34.462 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:34.462 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:34.462 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-osd.0.asok 2026-02-22T22:26:34.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:34.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-osd.0.asok dump_osd_network 2026-02-22T22:26:34.505 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:34.506 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 1000, 2026-02-22T22:26:34.506 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:26:34.506 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:34.510 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.510 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:34.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:34.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:34.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2026-02-22T22:26:34.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:34.516 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mgr.x.asok 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:34.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-mgr.x.asok dump_osd_network 2026-02-22T22:26:34.555 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:34.555 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 1000, 2026-02-22T22:26:34.555 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:26:34.555 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:34.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:34.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:34.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:34.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2026-02-22T22:26:34.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:34.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2026-02-22T22:26:34.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:26:34.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:26:34.566 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:34.566 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:34.566 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:34.566 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-osd.0.asok 2026-02-22T22:26:34.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:34.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-osd.0.asok dump_osd_network 0 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 0, 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [ 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:33 2026", 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.132, 2026-02-22T22:26:34.610 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.135, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.145 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.166, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.227, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.227 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.166, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.227, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.227 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.166 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:33 2026", 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.120, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.124, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.135 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.148, 2026-02-22T22:26:34.611 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.165, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.374 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.148, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.165, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.374 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.140 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:33 2026", 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.612 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.114, 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.106, 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.123 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.156, 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.156, 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.231 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.156, 2026-02-22T22:26:34.613 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.156, 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.231 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.156 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:33 2026", 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.614 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.100, 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.092, 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.105 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.125, 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.183, 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.313 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.615 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.125, 2026-02-22T22:26:34.616 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.183, 2026-02-22T22:26:34.616 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.313 2026-02-22T22:26:34.616 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.616 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.125 2026-02-22T22:26:34.616 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:26:34.616 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:26:34.616 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:34.617 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.617 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2026-02-22T22:26:34.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:34.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:34.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mgr.x.asok 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:26:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-mgr.x.asok dump_osd_network 0 2026-02-22T22:26:34.662 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:26:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 0, 2026-02-22T22:26:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [ 2026-02-22T22:26:34.662 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.128, 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.140, 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.201 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.121, 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.092, 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.092 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.150, 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.249, 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 2.029 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.191 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.132, 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.140, 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.162 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.114, 2026-02-22T22:26:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.097, 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.097 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.155, 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.263, 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.424 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.153 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.092, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.116, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.162 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.081, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.072, 2026-02-22T22:26:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.072 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.112, 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.250, 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 1.993 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.165 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.127, 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.118, 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.153 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.092, 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.088, 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.080 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.186, 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.186, 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.286 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.088 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.122, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.129, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.151 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.107, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.098, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.095 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.147, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.175, 2026-02-22T22:26:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.301 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.131 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.118, 2026-02-22T22:26:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.137, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.149 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.098, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.093, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.093 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.161, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.262, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.262 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.139 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.126, 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.133, 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.146 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.120, 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.109, 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.101 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.144, 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.227, 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.227 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.114 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.126, 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.121, 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.143 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.111, 2026-02-22T22:26:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.077, 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.077 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.142, 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.169, 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.280 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.112 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.110, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.107, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.141 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.089, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.086, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.084 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.166, 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.166, 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.321 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.074 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.124, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.122, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.136 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.112, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.103, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.089 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.138, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.165, 2026-02-22T22:26:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.374 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.107 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:26:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.104, 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.106, 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.124 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.088, 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.088, 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.078 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.135, 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.145, 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.231 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.096 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:26:28 2026", 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:26:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.089, 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.089, 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.105 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.083, 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.072, 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.067 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.095, 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.183, 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.313 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.086 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:26:34.685 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:26:34.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:26:34.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2026-02-22T22:26:34.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:26:34.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2026-02-22T22:26:34.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:26:34.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2026-02-22T22:27:24.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2026-02-22T22:27:24.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:27:24.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:27:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:27:24.866 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:27:24.866 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T22:27:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:27:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:27:24.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:27:24.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836510 2026-02-22T22:27:24.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836510 2026-02-22T22:27:24.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836510' 2026-02-22T22:27:24.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:27:24.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:27:24.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672989 2026-02-22T22:27:24.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672989 2026-02-22T22:27:24.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836510 1-42949672989' 2026-02-22T22:27:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:27:24.993 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:27:25.054 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509467 2026-02-22T22:27:25.054 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509467 2026-02-22T22:27:25.054 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509467' 2026-02-22T22:27:25.054 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:27:25.055 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836510 2026-02-22T22:27:25.055 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:27:25.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:27:25.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836510 2026-02-22T22:27:25.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:27:25.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836510 2026-02-22T22:27:25.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-22T22:27:25.058 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836510 2026-02-22T22:27:25.058 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:27:25.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836508 -lt 21474836510 2026-02-22T22:27:25.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:27:26.246 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:27:26.247 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:27:26.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836510 -lt 21474836510 2026-02-22T22:27:26.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:27:26.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672989 2026-02-22T22:27:26.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:27:26.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:27:26.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672989 2026-02-22T22:27:26.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:27:26.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672989 2026-02-22T22:27:26.439 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672989 2026-02-22T22:27:26.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2026-02-22T22:27:26.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:27:26.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672989 -lt 42949672989 2026-02-22T22:27:26.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:27:26.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509467 2026-02-22T22:27:26.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:27:26.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:27:26.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509467 2026-02-22T22:27:26.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:27:26.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509467 2026-02-22T22:27:26.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2026-02-22T22:27:26.630 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509467 2026-02-22T22:27:26.630 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:27:26.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509467 -lt 64424509467 2026-02-22T22:27:26.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-osd.0.asok 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:27:26.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-osd.0.asok dump_osd_network 2026-02-22T22:27:26.860 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:27:26.861 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 1000, 2026-02-22T22:27:26.861 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:27:26.861 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:27:26.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:26.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:27:26.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:27:26.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:26.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2026-02-22T22:27:26.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2026-02-22T22:27:26.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:27:26.871 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-22T22:27:26.871 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2026-02-22T22:27:26.872 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2026-02-22T22:27:26.872 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:26.872 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:26.872 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:26.872 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mgr.x.asok 2026-02-22T22:27:26.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:27:26.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-mgr.x.asok dump_osd_network 2026-02-22T22:27:26.910 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:27:26.910 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 1000, 2026-02-22T22:27:26.910 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:27:26.911 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:27:26.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:26.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:27:26.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:27:26.917 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:26.917 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2026-02-22T22:27:26.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2026-02-22T22:27:26.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:27:26.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2026-02-22T22:27:26.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:27:26.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:27:26.921 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:26.921 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:26.921 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:26.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-osd.0.asok 2026-02-22T22:27:26.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:27:26.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-osd.0.asok dump_osd_network 0 2026-02-22T22:27:26.963 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:27:26.963 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 0, 2026-02-22T22:27:26.963 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [ 2026-02-22T22:27:26.963 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:26.963 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.144, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.130, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.132 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.197, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.197, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.227 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.197, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.197, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.227 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.148 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.116, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.118, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.119 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.152, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.152, 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.165 2026-02-22T22:27:26.964 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.152, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.152, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.165 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.139 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.113, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.106, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.101 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.141, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.149, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.156 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.141, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.149, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.156 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.126 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.084, 2026-02-22T22:27:26.965 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.081, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.086 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.123, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.123, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.183 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.123, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.123, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.183 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.113 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:27:26.966 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:27:26.968 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:26.968 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:27:26.970 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2026-02-22T22:27:26.970 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:26.970 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2026-02-22T22:27:26.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:27:26.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:27:26.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-22T22:27:26.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2026-02-22T22:27:26.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2026-02-22T22:27:26.973 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:26.973 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:26.974 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:26.974 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mgr.x.asok 2026-02-22T22:27:26.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:27:26.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-mgr.x.asok dump_osd_network 0 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 0, 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [ 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:27:27.013 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.144, 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.130, 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.132 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.115, 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.087, 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.087 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.014 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.197, 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.197, 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.227 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.153 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:27.015 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.121, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.123, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.139 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.098, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.098, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.092 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.152, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.164, 2026-02-22T22:27:27.016 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.325 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.169 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.133, 2026-02-22T22:27:27.017 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.136, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.138 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.113, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.107, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.096 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.151, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.188, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.307 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.018 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.157 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.124, 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.115, 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.128 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.019 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.107, 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.086, 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.086 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.139, 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.199, 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.336 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.145 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.020 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.116, 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.118, 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.119 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.097, 2026-02-22T22:27:27.021 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.092, 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.092 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.152, 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.152, 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.165 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.122 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.022 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.099, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.109, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.117 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.083, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.075, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.075 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.023 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.114, 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.187, 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.187 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.114 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 2, 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:27:27.024 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.098, 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.099, 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.114 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.087, 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.083, 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.072 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.109, 2026-02-22T22:27:27.025 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.130, 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.270 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.136 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.026 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.113, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.106, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.101 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.088, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.080, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.076 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.141, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.149, 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.156 2026-02-22T22:27:27.027 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.141 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 2, 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.089, 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.101, 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.108 2026-02-22T22:27:27.028 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.076, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.069, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.069 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.106, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.141, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.187 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.103 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.029 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "front", 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.084, 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.098, 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.106 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.073, 2026-02-22T22:27:27.030 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.070, 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.070 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.098, 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.167, 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.186 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.095 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.031 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 1, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 0, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.101, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.106, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.103 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.091, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.081, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.071 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.032 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.121, 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.151, 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.190 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.088 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "last update": "Sun Feb 22 22:27:23 2026", 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "stale": false, 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "from osd": 0, 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "to osd": 1, 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "interface": "back", 2026-02-22T22:27:27.033 INFO:tasks.workunit.client.0.trial123.stdout: "average": { 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.084, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.081, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.086 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "min": { 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.060, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.060, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.060 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "max": { 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "1min": 0.123, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "5min": 0.123, 2026-02-22T22:27:27.034 INFO:tasks.workunit.client.0.trial123.stdout: "15min": 0.183 2026-02-22T22:27:27.035 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:27:27.035 INFO:tasks.workunit.client.0.trial123.stdout: "last": 0.133 2026-02-22T22:27:27.035 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:27:27.035 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:27:27.035 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:27:27.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:27.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2026-02-22T22:27:27.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2026-02-22T22:27:27.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:27.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2026-02-22T22:27:27.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2026-02-22T22:27:27.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mgr.x.asok 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:27:27.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-mgr.x.asok dump_osd_network 99 2026-02-22T22:27:27.063 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:27:27.063 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 99, 2026-02-22T22:27:27.063 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:27:27.063 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:27:27.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:27.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2026-02-22T22:27:27.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2026-02-22T22:27:27.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2026-02-22T22:27:27.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2026-02-22T22:27:27.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:27:27.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:27:27.071 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:27.071 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.071 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-osd.0.asok 2026-02-22T22:27:27.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2026-02-22T22:27:27.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.51626/ceph-osd.0.asok dump_osd_network 98 2026-02-22T22:27:27.113 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:27:27.114 INFO:tasks.workunit.client.0.trial123.stdout: "threshold": 98, 2026-02-22T22:27:27.114 INFO:tasks.workunit.client.0.trial123.stdout: "entries": [] 2026-02-22T22:27:27.114 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:27:27.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2026-02-22T22:27:27.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2026-02-22T22:27:27.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2026-02-22T22:27:27.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2026-02-22T22:27:27.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-22T22:27:27.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2026-02-22T22:27:27.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:27:27.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2026-02-22T22:27:27.122 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:27:27.122 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:27:27.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:27:27.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:27:27.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:27:27.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:27:27.229 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:27:27.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:27:27.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:27:27.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:27:27.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:27:27.231 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:27:27.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:27:27.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:27:27.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:27:27.233 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:27:27.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:27:27.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:27:27.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2026-02-22T22:27:27.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:27:27.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.51626 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:27:27.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2026-02-22T22:27:27.816 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:27:27.816 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:27:27.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:27:27.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:27:27.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:27:27.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:27:27.817 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:27:27.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:27:27.817 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:27:27.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:27:27.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:27:27.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:27:27.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:27:27.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:27:27.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:27:27.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:27:27.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:27:27.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:27:27.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2026-02-22T22:27:27.822 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:27:27.822 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.822 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.51626 2026-02-22T22:27:27.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:27:27.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:27:27.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2026-02-22T22:27:27.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:27:27.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.51626 2026-02-22T22:27:27.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:27:27.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:27:27.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:27:27.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-22T22:27:27.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2026-02-22T22:27:27.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2026-02-22T22:27:27.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2026-02-22T22:27:27.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-22T22:27:27.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:27:27.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:27:27.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:27:27.847 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:27:27.847 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.847 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.848 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:27.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:27:27.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:27:27.875 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:27:27.876 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:27.876 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.876 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.876 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mon.a.asok 2026-02-22T22:27:27.876 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:27:27.876 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.51626/ceph-mon.a.asok config get fsid 2026-02-22T22:27:27.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.51626/ceph-mon.a.asok 2026-02-22T22:27:27.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:27:27.920 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.51626/ceph-mon.a.asok config get mon_host 2026-02-22T22:27:27.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2026-02-22T22:27:27.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2026-02-22T22:27:27.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:27:27.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:27:27.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:27:27.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2026-02-22T22:27:27.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:27:28.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:27:28.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:27:28.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:27:28.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:27:28.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:28.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:28.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:28.062 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:27:28.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:27:28.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2026-02-22T22:27:28.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-22T22:27:28.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:27:28.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2026-02-22T22:27:28.084 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:27:28.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2072353-f4cd-43ba-888c-37c9253d0627 2026-02-22T22:27:28.085 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 b2072353-f4cd-43ba-888c-37c9253d0627 2026-02-22T22:27:28.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b2072353-f4cd-43ba-888c-37c9253d0627' 2026-02-22T22:27:28.086 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:27:28.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQgptp9laABRAAcp06Tn0uBv40FIEQdLXoWA== 2026-02-22T22:27:28.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQgptp9laABRAAcp06Tn0uBv40FIEQdLXoWA=="}' 2026-02-22T22:27:28.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2072353-f4cd-43ba-888c-37c9253d0627 -i td/network-ping/0/new.json 2026-02-22T22:27:28.196 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:27:28.203 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2026-02-22T22:27:28.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQgptp9laABRAAcp06Tn0uBv40FIEQdLXoWA== --osd-uuid b2072353-f4cd-43ba-888c-37c9253d0627 2026-02-22T22:27:28.222 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:28.220+0000 7fb90741e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:28.223 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:28.221+0000 7fb90741e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:28.225 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:28.222+0000 7fb90741e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:28.225 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:28.223+0000 7fb90741e780 -1 bdev(0x55fe85b6bc00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:27:28.225 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:28.223+0000 7fb90741e780 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-22T22:27:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-22T22:27:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:27:30.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:27:30.872 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:27:30.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:27:31.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:27:31.123 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:27:31.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:27:31.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:27:31.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:27:31.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:27:31.140 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:31.138+0000 7fdfe9f19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:31.141 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:31.139+0000 7fdfe9f19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:31.142 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:31.140+0000 7fdfe9f19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:31.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:27:31.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:27:31.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:27:31.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:27:31.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:27:31.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:31.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:27:31.311 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:27:31.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:31.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:27:31.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:32.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:32.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:32.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:27:32.502 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:27:32.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:32.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:27:32.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:32.736 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:32.734+0000 7fdfe9f19780 -1 Falling back to public interface 2026-02-22T22:27:33.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:33.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:33.689 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:27:33.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:27:33.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:33.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:27:33.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:34.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:34.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:34.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:27:34.881 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:27:34.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:34.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:27:35.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:35.671 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:35.669+0000 7fdfe9f19780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:27:36.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:36.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:36.071 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:27:36.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:27:36.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:36.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:27:36.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:37.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:37.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:37.265 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:27:37.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:27:37.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:37.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:27:37.453 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/667234584,v1:127.0.0.1:6803/667234584] [v2:127.0.0.1:6804/667234584,v1:127.0.0.1:6805/667234584] exists,up b2072353-f4cd-43ba-888c-37c9253d0627 2026-02-22T22:27:37.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-22T22:27:37.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:37.459 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:27:37.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:27:37.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:27:37.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2026-02-22T22:27:37.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:27:37.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e38b4a8-91da-407d-b2b4-bd2ef3fd2473 2026-02-22T22:27:37.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9e38b4a8-91da-407d-b2b4-bd2ef3fd2473' 2026-02-22T22:27:37.461 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 9e38b4a8-91da-407d-b2b4-bd2ef3fd2473 2026-02-22T22:27:37.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:27:37.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZgptpa6jYGxAAh4G6ATpVBfKzeetgJqwPcg== 2026-02-22T22:27:37.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZgptpa6jYGxAAh4G6ATpVBfKzeetgJqwPcg=="}' 2026-02-22T22:27:37.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e38b4a8-91da-407d-b2b4-bd2ef3fd2473 -i td/network-ping/1/new.json 2026-02-22T22:27:37.663 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:27:37.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2026-02-22T22:27:37.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZgptpa6jYGxAAh4G6ATpVBfKzeetgJqwPcg== --osd-uuid 9e38b4a8-91da-407d-b2b4-bd2ef3fd2473 2026-02-22T22:27:37.688 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:37.686+0000 7f8c00584780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:37.690 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:37.688+0000 7f8c00584780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:37.690 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:37.689+0000 7f8c00584780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:37.691 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:37.689+0000 7f8c00584780 -1 bdev(0x56380c187c00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:27:37.691 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:37.689+0000 7f8c00584780 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-22T22:27:40.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-22T22:27:40.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:27:40.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:27:40.052 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:27:40.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:27:40.308 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:27:40.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:27:40.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:27:40.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:27:40.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:27:40.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:27:40.326 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:40.325+0000 7f4649b53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:40.328 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:40.326+0000 7f4649b53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:40.329 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:40.327+0000 7f4649b53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:40.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:27:40.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:27:40.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:27:40.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:27:40.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:27:40.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:40.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:27:40.500 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:27:40.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:40.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:27:40.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:41.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:41.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:41.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:27:41.689 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:27:41.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:41.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:27:41.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:41.911 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:41.909+0000 7f4649b53780 -1 Falling back to public interface 2026-02-22T22:27:42.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:42.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:42.881 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:27:42.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:27:42.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:42.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:27:43.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:44.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:44.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:44.070 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:27:44.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:27:44.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:44.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:27:44.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:44.612 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:44.610+0000 7f4649b53780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:27:45.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:45.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:45.261 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:27:45.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:27:45.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:45.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:27:45.452 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/8240081,v1:127.0.0.1:6811/8240081] [v2:127.0.0.1:6812/8240081,v1:127.0.0.1:6813/8240081] exists,up 9e38b4a8-91da-407d-b2b4-bd2ef3fd2473 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:27:45.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:27:45.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:27:45.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bea2c865-09f3-4004-9cbb-1888b491cc92 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bea2c865-09f3-4004-9cbb-1888b491cc92' 2026-02-22T22:27:45.459 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 bea2c865-09f3-4004-9cbb-1888b491cc92 2026-02-22T22:27:45.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:27:45.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhgptpVaS/GxAAhGRYNh/oXtHjhyWTI3OPfA== 2026-02-22T22:27:45.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhgptpVaS/GxAAhGRYNh/oXtHjhyWTI3OPfA=="}' 2026-02-22T22:27:45.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bea2c865-09f3-4004-9cbb-1888b491cc92 -i td/network-ping/2/new.json 2026-02-22T22:27:45.665 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:27:45.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2026-02-22T22:27:45.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhgptpVaS/GxAAhGRYNh/oXtHjhyWTI3OPfA== --osd-uuid bea2c865-09f3-4004-9cbb-1888b491cc92 2026-02-22T22:27:45.691 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:45.689+0000 7f04e541f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:45.693 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:45.691+0000 7f04e541f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:45.694 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:45.692+0000 7f04e541f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:45.694 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:45.692+0000 7f04e541f780 -1 bdev(0x556019c1dc00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:27:45.694 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:45.692+0000 7f04e541f780 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-22T22:27:47.811 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-22T22:27:47.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:27:47.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:27:47.813 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:27:47.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:27:48.067 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:27:48.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:27:48.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=55935cad-450a-40df-a7bc-3b0f4dad52be --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.51626/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:27:48.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:27:48.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:27:48.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:27:48.085 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:48.083+0000 7f860ee3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:48.086 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:48.084+0000 7f860ee3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:48.087 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:48.085+0000 7f860ee3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:27:48.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:27:48.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:27:48.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:27:48.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:27:48.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:27:48.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:48.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:27:48.258 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:27:48.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:48.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:27:48.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:49.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:49.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:49.445 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:27:49.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:27:49.446 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:49.446 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:27:49.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:49.679 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:49.677+0000 7f860ee3f780 -1 Falling back to public interface 2026-02-22T22:27:50.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:50.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:50.634 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:27:50.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:27:50.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:50.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:27:50.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:51.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:51.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:51.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:27:51.820 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:27:51.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:51.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:27:52.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:52.611 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:27:52.607+0000 7f860ee3f780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:27:53.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:53.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:53.011 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:27:53.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:27:53.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:53.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:27:53.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:27:54.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:27:54.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:27:54.205 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:27:54.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:27:54.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:27:54.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:27:54.395 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3538462877,v1:127.0.0.1:6819/3538462877] [v2:127.0.0.1:6820/3538462877,v1:127.0.0.1:6821/3538462877] exists,up bea2c865-09f3-4004-9cbb-1888b491cc92 2026-02-22T22:27:54.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:27:54.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:27:54.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:27:54.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2026-02-22T22:27:59.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2026-02-22T22:27:59.636 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'dc1' already exists 2026-02-22T22:27:59.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2026-02-22T22:27:59.840 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'dc2' already exists 2026-02-22T22:27:59.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2026-02-22T22:28:00.042 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'dc3' already exists 2026-02-22T22:28:00.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2026-02-22T22:28:00.245 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'rack1' already exists 2026-02-22T22:28:00.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2026-02-22T22:28:00.447 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'rack2' already exists 2026-02-22T22:28:00.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2026-02-22T22:28:00.651 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'rack3' already exists 2026-02-22T22:28:00.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2026-02-22T22:28:00.853 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'host1' already exists 2026-02-22T22:28:00.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2026-02-22T22:28:01.056 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'host2' already exists 2026-02-22T22:28:01.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2026-02-22T22:28:01.257 INFO:tasks.workunit.client.0.trial123.stderr:bucket 'host3' already exists 2026-02-22T22:28:01.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2026-02-22T22:28:01.460 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2026-02-22T22:28:01.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2026-02-22T22:28:01.663 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2026-02-22T22:28:01.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2026-02-22T22:28:01.865 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2026-02-22T22:28:01.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2026-02-22T22:28:02.068 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2026-02-22T22:28:02.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2026-02-22T22:28:02.271 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2026-02-22T22:28:02.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2026-02-22T22:28:02.474 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2026-02-22T22:28:02.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2026-02-22T22:28:02.677 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2026-02-22T22:28:02.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2026-02-22T22:28:02.879 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2026-02-22T22:28:02.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2026-02-22T22:28:03.081 INFO:tasks.workunit.client.0.trial123.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2026-02-22T22:28:03.089 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2026-02-22T22:28:03.284 INFO:tasks.workunit.client.0.trial123.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2026-02-22T22:28:03.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2026-02-22T22:28:03.485 INFO:tasks.workunit.client.0.trial123.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2026-02-22T22:28:03.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2026-02-22T22:28:03.689 INFO:tasks.workunit.client.0.trial123.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2026-02-22T22:28:03.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2026-02-22T22:28:03.891 INFO:tasks.workunit.client.0.trial123.stderr:rule myrule already exists 2026-02-22T22:28:03.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2026-02-22T22:28:03.899 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2026-02-22T22:28:04.146 INFO:tasks.workunit.client.0.trial123.stderr:pool 'foo' already exists 2026-02-22T22:28:04.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:28:05.157 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-22T22:28:05.177 INFO:tasks.workunit.client.0.trial123.stdout:hints = 1 2026-02-22T22:28:05.177 INFO:tasks.workunit.client.0.trial123.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-22T22:28:05.177 INFO:tasks.workunit.client.0.trial123.stdout:Object prefix: benchmark_data_trial123_62513 2026-02-22T22:28:05.178 INFO:tasks.workunit.client.0.trial123.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-22T22:28:05.178 INFO:tasks.workunit.client.0.trial123.stdout: 0 0 0 0 0 0 - 0 2026-02-22T22:28:06.177 INFO:tasks.workunit.client.0.trial123.stdout: 1 16 10883 10867 42.446 42.4492 0.00119714 0.00147016 2026-02-22T22:28:07.177 INFO:tasks.workunit.client.0.trial123.stdout: 2 16 22087 22071 43.1041 43.7656 0.00157181 0.00144816 2026-02-22T22:28:08.177 INFO:tasks.workunit.client.0.trial123.stdout: 3 16 34031 34015 44.2869 46.6562 0.00424798 0.00140939 2026-02-22T22:28:09.177 INFO:tasks.workunit.client.0.trial123.stdout: 4 16 45861 45845 44.7671 46.2109 0.00111886 0.00139461 2026-02-22T22:28:10.177 INFO:tasks.workunit.client.0.trial123.stdout: 5 16 57352 57336 44.7903 44.8867 0.00151009 0.00139388 2026-02-22T22:28:11.177 INFO:tasks.workunit.client.0.trial123.stdout: 6 16 66919 66903 43.5533 37.3711 0.00149008 0.00143354 2026-02-22T22:28:12.178 INFO:tasks.workunit.client.0.trial123.stdout: 7 16 76413 76397 42.629 37.0859 0.0019396 0.00146468 2026-02-22T22:28:13.178 INFO:tasks.workunit.client.0.trial123.stdout: 8 16 85816 85800 41.8913 36.7305 0.00135758 0.00149049 2026-02-22T22:28:14.178 INFO:tasks.workunit.client.0.trial123.stdout: 9 16 95041 95025 41.2403 36.0352 0.00175802 0.00151406 2026-02-22T22:28:15.178 INFO:tasks.workunit.client.0.trial123.stdout: 10 2 104448 104446 40.7961 36.8008 0.00195061 0.00153064 2026-02-22T22:28:16.178 INFO:tasks.workunit.client.0.trial123.stdout:Total time run: 10.0008 2026-02-22T22:28:16.178 INFO:tasks.workunit.client.0.trial123.stdout:Total writes made: 104448 2026-02-22T22:28:16.178 INFO:tasks.workunit.client.0.trial123.stdout:Write size: 4096 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Object size: 4096 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Bandwidth (MB/sec): 40.7968 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Stddev Bandwidth: 4.37925 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Max bandwidth (MB/sec): 46.6562 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Min bandwidth (MB/sec): 36.0352 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Average IOPS: 10443 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Stddev IOPS: 1121.09 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Max IOPS: 11944 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Min IOPS: 9225 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Average Latency(s): 0.00153065 2026-02-22T22:28:16.179 INFO:tasks.workunit.client.0.trial123.stdout:Stddev Latency(s): 0.000607603 2026-02-22T22:28:16.180 INFO:tasks.workunit.client.0.trial123.stdout:Max latency(s): 0.0106748 2026-02-22T22:28:16.180 INFO:tasks.workunit.client.0.trial123.stdout:Min latency(s): 0.000468007 2026-02-22T22:28:16.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2026-02-22T22:28:26.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2026-02-22T22:28:26.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:28:26.185 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:28:26.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:28:26.377 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:28:26.377 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T22:28:26.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:28:26.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:28:26.377 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:28:26.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836492 2026-02-22T22:28:26.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836492 2026-02-22T22:28:26.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836492' 2026-02-22T22:28:26.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:28:26.441 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:28:26.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=38654705675 2026-02-22T22:28:26.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 38654705675 2026-02-22T22:28:26.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836492 1-38654705675' 2026-02-22T22:28:26.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:28:26.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:28:26.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542153 2026-02-22T22:28:26.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542153 2026-02-22T22:28:26.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836492 1-38654705675 2-60129542153' 2026-02-22T22:28:26.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:28:26.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:28:26.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836492 2026-02-22T22:28:26.567 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:28:26.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836492 2026-02-22T22:28:26.568 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:28:26.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836492 2026-02-22T22:28:26.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2026-02-22T22:28:26.569 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836492 2026-02-22T22:28:26.569 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:28:26.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836490 -lt 21474836492 2026-02-22T22:28:26.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:28:27.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:28:27.756 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:28:27.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836490 -lt 21474836492 2026-02-22T22:28:27.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:28:28.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:28:28.948 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:28:29.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836493 -lt 21474836492 2026-02-22T22:28:29.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:28:29.138 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-38654705675 2026-02-22T22:28:29.138 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:28:29.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:28:29.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-38654705675 2026-02-22T22:28:29.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:28:29.140 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=38654705675 2026-02-22T22:28:29.140 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2026-02-22T22:28:29.141 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 38654705675 2026-02-22T22:28:29.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:28:29.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 38654705675 -lt 38654705675 2026-02-22T22:28:29.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:28:29.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542153 2026-02-22T22:28:29.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:28:29.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:28:29.329 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542153 2026-02-22T22:28:29.329 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:28:29.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542153 2026-02-22T22:28:29.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-22T22:28:29.331 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 60129542153 2026-02-22T22:28:29.331 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:28:29.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-22T22:28:29.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2026-02-22T22:28:29.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2026-02-22T22:28:29.735 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.177ms); Slow OSD heartbeats on front (longest 0.194ms); 1 pool(s) do not have an application enabled 2026-02-22T22:28:29.742 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2026-02-22T22:28:29.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2026-02-22T22:28:29.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2026-02-22T22:28:29.996 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.177ms); Slow OSD heartbeats on front (longest 0.194ms); 1 pool(s) do not have an application enabled 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.177ms) 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.177 msec 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.159 msec 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.153 msec 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.137 msec possibly improving 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.130 msec 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.105 msec 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.194ms) 2026-02-22T22:28:29.997 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.194 msec 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.169 msec 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.151 msec 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.147 msec 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.140 msec 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.135 msec 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout: application not enabled on pool 'foo' 2026-02-22T22:28:29.998 INFO:tasks.workunit.client.0.trial123.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2026-02-22T22:28:30.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2026-02-22T22:28:30.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2026-02-22T22:28:30.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2026-02-22T22:28:30.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2026-02-22T22:28:30.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:28:30.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:28:30.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:28:30.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:28:30.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:28:30.117 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:28:30.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:28:30.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:28:30.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:28:30.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:30.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:28:30.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:28:30.120 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:30.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:28:30.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:28:30.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2026-02-22T22:28:30.716 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:28:30.716 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:30.717 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:28:30.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.51626 2026-02-22T22:28:30.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:28:30.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:28:30.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/network-ping 0 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:28:30.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:28:30.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:28:30.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:28:30.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:28:30.721 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:28:30.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:28:30.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:28:30.722 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:28:30.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:30.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:28:30.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:28:30.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:30.724 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:28:30.725 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T22:28:30.725 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2026-02-22T22:28:30.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:28:30.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:30.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.51626 2026-02-22T22:28:30.726 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.51626 2026-02-22T22:28:30.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:28:30.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:28:30.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T22:28:30.728 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T22:28:30.728 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T22:28:30.794 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2026-02-22T22:28:30.795 DEBUG:teuthology.orchestra.run.trial123:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2026-02-22T22:28:30.859 INFO:tasks.workunit.client.0.trial123.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T22:28:30.864 INFO:tasks.workunit.client.0.trial123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T22:28:30.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:28:30.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:28:30.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T22:28:30.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T22:28:30.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T22:28:30.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/ok-to-stop 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2026-02-22T22:28:30.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none ' 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none ' 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none ' 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2026-02-22T22:28:30.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:TEST_0_osd 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:TEST_1_mon_checks 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:TEST_2_mons_checks 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:TEST_3_mons_checks 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:TEST_4_mons_checks 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:TEST_5_mons_checks' 2026-02-22T22:28:30.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:28:30.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:28:30.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:28:30.872 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:28:30.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:28:30.872 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:28:30.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:28:30.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:28:30.872 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:28:30.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:30.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:28:30.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:28:30.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:28:30.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2026-02-22T22:28:30.876 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:28:30.876 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:30.876 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:30.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62918 2026-02-22T22:28:30.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:28:30.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T22:28:30.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T22:28:30.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:28:30.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-22T22:28:30.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2026-02-22T22:28:30.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-22T22:28:30.897 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:28:30.897 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:30.897 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:30.897 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:30.897 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:30.897 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:30.898 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:30.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-22T22:28:30.923 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:28:30.925 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:28:30.926 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:30.926 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:30.926 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:28:30.926 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:28:30.926 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get fsid 2026-02-22T22:28:30.971 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:28:30.971 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:28:30.971 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:28:30.971 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:28:30.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get mon_host 2026-02-22T22:28:31.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2026-02-22T22:28:31.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2026-02-22T22:28:31.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:28:31.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:28:31.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:28:31.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2026-02-22T22:28:31.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:28:31.114 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:28:31.114 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:31.114 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:31.114 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:31.114 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:31.114 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:31.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:31.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:28:31.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:28:31.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:31.136 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:28:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:28:31.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2026-02-22T22:28:31.138 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:28:31.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1fed21a5-54ee-4f7f-8390-3547585e3408 2026-02-22T22:28:31.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1fed21a5-54ee-4f7f-8390-3547585e3408' 2026-02-22T22:28:31.138 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 1fed21a5-54ee-4f7f-8390-3547585e3408 2026-02-22T22:28:31.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:28:31.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPg5tp3gSRCBAAei54S1cKRv2Fh4QdnKOPig== 2026-02-22T22:28:31.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPg5tp3gSRCBAAei54S1cKRv2Fh4QdnKOPig=="}' 2026-02-22T22:28:31.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1fed21a5-54ee-4f7f-8390-3547585e3408 -i td/ok-to-stop/0/new.json 2026-02-22T22:28:31.245 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:28:31.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2026-02-22T22:28:31.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPg5tp3gSRCBAAei54S1cKRv2Fh4QdnKOPig== --osd-uuid 1fed21a5-54ee-4f7f-8390-3547585e3408 2026-02-22T22:28:31.268 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:31.265+0000 7fb50549d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:31.269 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:31.267+0000 7fb50549d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:31.270 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:31.268+0000 7fb50549d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:31.271 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:31.268+0000 7fb50549d780 -1 bdev(0x5605c634bc00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:28:31.271 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:31.268+0000 7fb50549d780 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-22T22:28:33.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-22T22:28:33.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:28:33.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:28:33.930 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:28:33.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:28:34.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:28:34.182 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:28:34.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:28:34.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:28:34.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:28:34.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:28:34.198 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:34.195+0000 7f9a61210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:34.199 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:34.196+0000 7f9a61210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:34.200 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:34.197+0000 7f9a61210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:34.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:28:34.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:28:34.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:28:34.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:28:34.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:28:34.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:34.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:28:34.369 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:28:34.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:34.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:34.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:28:35.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:28:35.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:35.555 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:28:35.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:28:35.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:35.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:35.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:28:35.794 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:35.792+0000 7f9a61210780 -1 Falling back to public interface 2026-02-22T22:28:36.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:28:36.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:36.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:28:36.746 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:28:36.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:36.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:36.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:28:37.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:28:37.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:37.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:28:37.936 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:28:37.936 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:37.936 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:38.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:28:38.747 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:38.745+0000 7f9a61210780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:28:39.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:28:39.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:39.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:28:39.123 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:28:39.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:39.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:39.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:28:40.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:28:40.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:40.315 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:28:40.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:28:40.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:40.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:40.505 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3513005059,v1:127.0.0.1:6803/3513005059] [v2:127.0.0.1:6804/3513005059,v1:127.0.0.1:6805/3513005059] exists,up 1fed21a5-54ee-4f7f-8390-3547585e3408 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2026-02-22T22:28:40.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:40.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-22T22:28:40.523 INFO:tasks.workunit.client.0.trial123.stdout:starting mds.a at 2026-02-22T22:28:40.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2026-02-22T22:28:40.779 INFO:tasks.workunit.client.0.trial123.stderr:pool 'meta' already exists 2026-02-22T22:28:40.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2026-02-22T22:28:40.980 INFO:tasks.workunit.client.0.trial123.stderr:pool 'data' already exists 2026-02-22T22:28:40.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2026-02-22T22:28:41.340 INFO:tasks.workunit.client.0.trial123.stderr:filesystem 'myfs' already exists 2026-02-22T22:28:41.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2026-02-22T22:28:46.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2026-02-22T22:28:46.518 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4129 and/or standby gids 2026-02-22T22:28:46.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2026-02-22T22:28:46.685 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4129 and/or standby gids 2026-02-22T22:28:46.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2026-02-22T22:28:46.935 INFO:tasks.workunit.client.0.trial123.stderr:should be safe to stop [dne] 2026-02-22T22:28:46.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2026-02-22T22:28:46.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2026-02-22T22:28:46.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2026-02-22T22:28:46.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2026-02-22T22:28:46.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2026-02-22T22:28:46.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:46.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-22T22:28:46.959 INFO:tasks.workunit.client.0.trial123.stdout:starting mds.b at 2026-02-22T22:28:46.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2026-02-22T22:28:51.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2026-02-22T22:28:52.211 INFO:tasks.workunit.client.0.trial123.stderr:should be safe to stop [a] 2026-02-22T22:28:52.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2026-02-22T22:28:52.467 INFO:tasks.workunit.client.0.trial123.stderr:should be safe to stop [b] 2026-02-22T22:28:52.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2026-02-22T22:28:52.643 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4129 and/or standby gids 4143 2026-02-22T22:28:52.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2026-02-22T22:28:52.889 INFO:tasks.workunit.client.0.trial123.stderr:should be safe to stop [a,dne1,dne2] 2026-02-22T22:28:52.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2026-02-22T22:28:53.142 INFO:tasks.workunit.client.0.trial123.stderr:should be safe to stop [b,dne] 2026-02-22T22:28:53.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2026-02-22T22:28:53.315 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4129 and/or standby gids 4143 2026-02-22T22:28:53.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2026-02-22T22:28:53.573 INFO:tasks.workunit.client.0.trial123.stderr:should be safe to stop [dne1,dne2] 2026-02-22T22:28:53.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2026-02-22T22:28:53.581 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:28:53.581 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:28:53.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:28:53.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:28:53.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:28:53.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:28:53.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-22T22:28:53.688 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:28:53.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:28:53.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:28:53.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:28:53.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:28:53.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:28:53.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-22T22:28:53.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:28:53.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:28:53.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:28:53.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:28:53.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:28:53.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:28:53.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:28:53.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:28:53.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:28:53.803 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:28:53.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:28:53.804 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:28:53.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:28:53.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:28:53.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:28:53.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:53.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:28:53.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:28:53.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:53.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:28:53.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:28:53.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:28:53.812 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:28:53.812 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:53.812 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:53.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:28:53.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:28:53.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:28:53.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-22T22:28:53.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-22T22:28:53.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2026-02-22T22:28:53.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:28:53.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:28:53.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:28:53.816 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:28:53.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:28:53.817 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:28:53.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:28:53.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:28:53.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:28:53.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:53.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:28:53.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:28:53.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:28:53.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:28:53.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:28:53.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:28:53.822 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:28:53.822 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:53.822 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:53.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:28:53.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:28:53.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:28:53.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2026-02-22T22:28:53.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:28:53.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:53.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:53.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62918 2026-02-22T22:28:53.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:28:53.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:28:53.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2026-02-22T22:28:53.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-22T22:28:53.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:28:53.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:53.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:53.847 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:53.847 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:53.848 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:53.848 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:53.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-22T22:28:53.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:28:53.873 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:28:53.874 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:28:53.875 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:53.875 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:53.875 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:28:53.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:28:53.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get fsid 2026-02-22T22:28:53.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:28:53.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:28:53.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:28:53.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:28:53.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:28:53.920 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get mon_host 2026-02-22T22:28:53.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2026-02-22T22:28:53.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2026-02-22T22:28:53.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:28:53.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:28:53.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:28:53.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2026-02-22T22:28:53.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:54.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:28:54.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:28:54.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2026-02-22T22:28:54.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2026-02-22T22:28:54.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:28:54.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:28:54.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:28:54.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:28:54.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:28:54.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2026-02-22T22:28:54.082 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:28:54.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ef69fa1-8328-417f-8e18-86e5ec8a6fc8 2026-02-22T22:28:54.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6ef69fa1-8328-417f-8e18-86e5ec8a6fc8' 2026-02-22T22:28:54.082 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 6ef69fa1-8328-417f-8e18-86e5ec8a6fc8 2026-02-22T22:28:54.082 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:28:54.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmg5tpgA9XBRAAtyy3yMMRpGqwC4k9xTQ0/A== 2026-02-22T22:28:54.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmg5tpgA9XBRAAtyy3yMMRpGqwC4k9xTQ0/A=="}' 2026-02-22T22:28:54.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ef69fa1-8328-417f-8e18-86e5ec8a6fc8 -i td/ok-to-stop/0/new.json 2026-02-22T22:28:54.193 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:28:54.200 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2026-02-22T22:28:54.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmg5tpgA9XBRAAtyy3yMMRpGqwC4k9xTQ0/A== --osd-uuid 6ef69fa1-8328-417f-8e18-86e5ec8a6fc8 2026-02-22T22:28:54.218 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:54.216+0000 7f862182c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:54.219 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:54.217+0000 7f862182c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:54.220 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:54.218+0000 7f862182c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:54.220 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:54.218+0000 7f862182c780 -1 bdev(0x5629e9a13c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:28:54.221 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:54.218+0000 7f862182c780 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-22T22:28:57.104 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-22T22:28:57.104 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:28:57.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:28:57.105 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:28:57.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:28:57.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:28:57.360 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:28:57.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:28:57.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:28:57.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:28:57.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:28:57.375 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:57.373+0000 7f7f7ce18780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:57.376 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:57.374+0000 7f7f7ce18780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:57.377 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:57.375+0000 7f7f7ce18780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:28:57.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:28:57.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:28:57.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:28:57.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:28:57.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:28:57.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:57.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:28:57.547 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:28:57.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:57.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:57.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:28:58.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:28:58.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:58.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:28:58.731 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:28:58.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:58.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:28:58.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:28:58.955 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:28:58.953+0000 7f7f7ce18780 -1 Falling back to public interface 2026-02-22T22:28:59.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:28:59.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:28:59.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:28:59.919 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:28:59.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:28:59.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:29:00.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:01.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:01.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:01.113 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:29:01.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:29:01.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:01.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:29:01.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:01.888 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:01.886+0000 7f7f7ce18780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:29:02.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:02.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:02.302 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:29:02.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:29:02.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:02.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:29:02.495 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:03.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:03.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:03.497 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:29:03.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:29:03.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:03.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:29:03.684 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3974547006,v1:127.0.0.1:6803/3974547006] [v2:127.0.0.1:6804/3974547006,v1:127.0.0.1:6805/3974547006] exists,up 6ef69fa1-8328-417f-8e18-86e5ec8a6fc8 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:29:03.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:29:03.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2026-02-22T22:29:03.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:29:03.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=111ff596-c38c-4e5d-9957-78198733f3d5 2026-02-22T22:29:03.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 111ff596-c38c-4e5d-9957-78198733f3d5' 2026-02-22T22:29:03.688 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 111ff596-c38c-4e5d-9957-78198733f3d5 2026-02-22T22:29:03.688 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:29:03.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvg5tpJ8KOKRAAjQolmUCxL9v7XcvA+8+e1g== 2026-02-22T22:29:03.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvg5tpJ8KOKRAAjQolmUCxL9v7XcvA+8+e1g=="}' 2026-02-22T22:29:03.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 111ff596-c38c-4e5d-9957-78198733f3d5 -i td/ok-to-stop/1/new.json 2026-02-22T22:29:03.890 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:29:03.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2026-02-22T22:29:03.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvg5tpJ8KOKRAAjQolmUCxL9v7XcvA+8+e1g== --osd-uuid 111ff596-c38c-4e5d-9957-78198733f3d5 2026-02-22T22:29:03.914 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:03.911+0000 7f539e812780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:03.915 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:03.912+0000 7f539e812780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:03.916 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:03.913+0000 7f539e812780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:03.916 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:03.914+0000 7f539e812780 -1 bdev(0x564309b8bc00 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:29:03.916 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:03.914+0000 7f539e812780 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2026-02-22T22:29:06.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2026-02-22T22:29:06.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:29:06.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:29:06.026 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:29:06.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:29:06.280 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:29:06.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:29:06.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:29:06.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:29:06.282 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:29:06.282 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:29:06.296 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:06.293+0000 7fa344c1a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:06.297 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:06.294+0000 7fa344c1a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:06.298 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:06.295+0000 7fa344c1a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:06.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:29:06.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:29:06.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:29:06.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:29:06.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:29:06.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:06.469 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:29:06.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:29:06.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:06.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:29:06.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:07.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:07.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:07.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:29:07.660 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:29:07.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:07.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:29:07.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:07.883 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:07.880+0000 7fa344c1a780 -1 Falling back to public interface 2026-02-22T22:29:08.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:08.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:08.850 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:29:08.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:29:08.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:08.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:29:09.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:10.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:10.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:10.038 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:29:10.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:29:10.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:10.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:29:10.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:10.849 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:10.846+0000 7fa344c1a780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:29:11.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:11.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:11.224 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:29:11.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:29:11.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:11.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:29:11.412 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:12.414 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:12.414 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:12.415 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:29:12.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:29:12.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:12.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:29:12.599 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2828261462,v1:127.0.0.1:6811/2828261462] [v2:127.0.0.1:6812/2828261462,v1:127.0.0.1:6813/2828261462] exists,up 111ff596-c38c-4e5d-9957-78198733f3d5 2026-02-22T22:29:12.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:29:12.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:29:12.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:29:12.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2026-02-22T22:29:12.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:29:12.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d416c29e-575f-40e4-8782-3915d4e05a1f 2026-02-22T22:29:12.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d416c29e-575f-40e4-8782-3915d4e05a1f' 2026-02-22T22:29:12.607 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 d416c29e-575f-40e4-8782-3915d4e05a1f 2026-02-22T22:29:12.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:29:12.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4g5tpZtl0JBAAxOeN56VqTGt/utUC6f22Vg== 2026-02-22T22:29:12.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4g5tpZtl0JBAAxOeN56VqTGt/utUC6f22Vg=="}' 2026-02-22T22:29:12.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d416c29e-575f-40e4-8782-3915d4e05a1f -i td/ok-to-stop/2/new.json 2026-02-22T22:29:12.805 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:29:12.811 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2026-02-22T22:29:12.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4g5tpZtl0JBAAxOeN56VqTGt/utUC6f22Vg== --osd-uuid d416c29e-575f-40e4-8782-3915d4e05a1f 2026-02-22T22:29:12.829 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:12.826+0000 7f8cf2413780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:12.830 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:12.827+0000 7f8cf2413780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:12.831 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:12.829+0000 7f8cf2413780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:12.832 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:12.829+0000 7f8cf2413780 -1 bdev(0x55982e9f5c00 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:29:12.832 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:12.829+0000 7f8cf2413780 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2026-02-22T22:29:14.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2026-02-22T22:29:14.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:29:14.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:29:14.962 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:29:14.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:29:15.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:29:15.220 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:29:15.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:29:15.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:29:15.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:29:15.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:29:15.236 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:15.234+0000 7fc36740d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:15.237 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:15.236+0000 7fc36740d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:15.238 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:15.237+0000 7fc36740d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:29:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:29:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:29:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:29:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:29:15.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:15.411 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:29:15.412 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:29:15.412 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:15.412 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:29:15.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:16.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:16.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:16.601 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:29:16.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:29:16.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:16.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:29:16.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:16.824 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:16.822+0000 7fc36740d780 -1 Falling back to public interface 2026-02-22T22:29:17.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:17.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:17.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:29:17.799 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:29:17.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:17.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:29:17.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:18.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:18.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:18.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:29:18.987 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:29:18.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:18.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:29:19.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:19.765 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:19.764+0000 7fc36740d780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:29:20.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:20.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:20.177 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:29:20.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:29:20.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:20.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:29:20.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:21.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:21.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:21.372 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:29:21.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:29:21.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:21.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:29:21.557 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1666011320,v1:127.0.0.1:6819/1666011320] [v2:127.0.0.1:6820/1666011320,v1:127.0.0.1:6821/1666011320] exists,up d416c29e-575f-40e4-8782-3915d4e05a1f 2026-02-22T22:29:21.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:29:21.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:29:21.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:29:21.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2026-02-22T22:29:21.564 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:29:21.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=53e8c30b-c791-43f4-a77b-838fc85e5135 2026-02-22T22:29:21.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 53e8c30b-c791-43f4-a77b-838fc85e5135' 2026-02-22T22:29:21.565 INFO:tasks.workunit.client.0.trial123.stdout:add osd3 53e8c30b-c791-43f4-a77b-838fc85e5135 2026-02-22T22:29:21.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:29:21.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBg5tp8RH6IRAAj+C31qsOlLHhRGJIXzZpLQ== 2026-02-22T22:29:21.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBg5tp8RH6IRAAj+C31qsOlLHhRGJIXzZpLQ=="}' 2026-02-22T22:29:21.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 53e8c30b-c791-43f4-a77b-838fc85e5135 -i td/ok-to-stop/3/new.json 2026-02-22T22:29:21.766 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:29:21.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2026-02-22T22:29:21.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBg5tp8RH6IRAAj+C31qsOlLHhRGJIXzZpLQ== --osd-uuid 53e8c30b-c791-43f4-a77b-838fc85e5135 2026-02-22T22:29:21.790 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:21.788+0000 7f75d4418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:21.791 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:21.789+0000 7f75d4418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:21.792 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:21.790+0000 7f75d4418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:21.793 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:21.791+0000 7f75d4418780 -1 bdev(0x56355d777c00 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:29:21.793 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:21.791+0000 7f75d4418780 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2026-02-22T22:29:23.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2026-02-22T22:29:23.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:29:23.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:29:23.904 INFO:tasks.workunit.client.0.trial123.stdout:adding osd3 key to auth repository 2026-02-22T22:29:23.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:29:24.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:29:24.161 INFO:tasks.workunit.client.0.trial123.stdout:start osd.3 2026-02-22T22:29:24.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:29:24.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:29:24.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:29:24.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:29:24.176 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:24.174+0000 7f581d20e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:24.177 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:24.176+0000 7f581d20e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:24.178 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:24.177+0000 7f581d20e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:29:24.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:29:24.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:29:24.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:29:24.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:29:24.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:29:24.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:24.352 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:29:24.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:29:24.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:24.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:29:24.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:25.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:25.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:25.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:29:25.539 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:29:25.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:25.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:29:25.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:25.761 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:25.759+0000 7f581d20e780 -1 Falling back to public interface 2026-02-22T22:29:26.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:26.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:26.730 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:29:26.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:29:26.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:26.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:29:26.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:27.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:27.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:27.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:29:27.920 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:29:27.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:27.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:29:28.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:28.968 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:29:28.966+0000 7f581d20e780 -1 osd.3 0 log_to_monitors true 2026-02-22T22:29:29.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:29.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:29.108 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:29:29.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:29:29.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:29.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:29:29.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:29:30.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:29:30.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:29:30.302 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:29:30.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:29:30.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:29:30.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:29:30.488 INFO:tasks.workunit.client.0.trial123.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2674248150,v1:127.0.0.1:6827/2674248150] [v2:127.0.0.1:6828/2674248150,v1:127.0.0.1:6829/2674248150] exists,up 53e8c30b-c791-43f4-a77b-838fc85e5135 2026-02-22T22:29:30.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:29:30.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:29:30.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:29:30.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2026-02-22T22:29:30.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2026-02-22T22:29:31.035 INFO:tasks.workunit.client.0.trial123.stderr:pool 'ec' already exists 2026-02-22T22:29:31.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2026-02-22T22:29:31.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:29:31.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:29:31.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:29:31.044 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:29:31.045 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:29:31.045 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:29:31.046 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:29:31.046 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:29:31.046 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:29:31.133 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:29:31.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:29:31.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:29:31.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:29:31.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:29:31.134 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:29:31.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:29:31.323 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:29:31.324 INFO:tasks.workunit.client.0.trial123.stderr:2 2026-02-22T22:29:31.324 INFO:tasks.workunit.client.0.trial123.stderr:3' 2026-02-22T22:29:31.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:29:31.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:31.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:29:31.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836488 2026-02-22T22:29:31.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836488 2026-02-22T22:29:31.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836488' 2026-02-22T22:29:31.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:31.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:29:31.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672966 2026-02-22T22:29:31.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672966 2026-02-22T22:29:31.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2026-02-22T22:29:31.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:31.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:29:31.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509445 2026-02-22T22:29:31.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509445 2026-02-22T22:29:31.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509445' 2026-02-22T22:29:31.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:31.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:29:31.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=81604378627 2026-02-22T22:29:31.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 81604378627 2026-02-22T22:29:31.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509445 3-81604378627' 2026-02-22T22:29:31.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:31.571 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836488 2026-02-22T22:29:31.571 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:31.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:29:31.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836488 2026-02-22T22:29:31.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:31.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836488 2026-02-22T22:29:31.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-22T22:29:31.573 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836488 2026-02-22T22:29:31.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:29:31.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836486 -lt 21474836488 2026-02-22T22:29:31.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:29:32.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:29:32.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:29:32.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-22T22:29:32.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:32.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672966 2026-02-22T22:29:32.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:32.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:29:32.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672966 2026-02-22T22:29:32.953 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:32.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672966 2026-02-22T22:29:32.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-22T22:29:32.954 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672966 2026-02-22T22:29:32.954 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:29:33.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672967 -lt 42949672966 2026-02-22T22:29:33.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:33.147 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509445 2026-02-22T22:29:33.147 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:33.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:29:33.148 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509445 2026-02-22T22:29:33.148 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:33.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509445 2026-02-22T22:29:33.149 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509445 2026-02-22T22:29:33.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2026-02-22T22:29:33.150 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:29:33.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509445 -lt 64424509445 2026-02-22T22:29:33.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:33.336 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-81604378627 2026-02-22T22:29:33.336 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:33.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:29:33.337 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-81604378627 2026-02-22T22:29:33.337 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:33.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=81604378627 2026-02-22T22:29:33.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 81604378627' 2026-02-22T22:29:33.338 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.3 seq 81604378627 2026-02-22T22:29:33.338 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:29:33.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 81604378627 -lt 81604378627 2026-02-22T22:29:33.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:29:33.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:33.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:33.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 == 0 2026-02-22T22:29:33.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:29:33.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:29:33.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:29:33.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:29:33.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:29:33.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:29:33.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:29:33.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=32 2026-02-22T22:29:33.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:29:33.963 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:33.963 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:34.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 32 = 32 2026-02-22T22:29:34.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:29:34.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:29:34.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2026-02-22T22:29:34.498 INFO:tasks.workunit.client.0.trial123.stderr:set pool 1 min_size to 3 2026-02-22T22:29:34.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2026-02-22T22:29:34.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:29:34.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:29:34.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:29:34.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:29:34.511 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:29:34.511 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:29:34.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:29:34.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:29:34.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:29:34.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:29:34.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:29:34.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:29:34.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:29:34.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:29:34.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:29:34.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:29:34.783 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:29:34.783 INFO:tasks.workunit.client.0.trial123.stderr:2 2026-02-22T22:29:34.783 INFO:tasks.workunit.client.0.trial123.stderr:3' 2026-02-22T22:29:34.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:29:34.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:34.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:29:34.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836491 2026-02-22T22:29:34.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836491 2026-02-22T22:29:34.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T22:29:34.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:34.845 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:29:34.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672969 2026-02-22T22:29:34.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672969 2026-02-22T22:29:34.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2026-02-22T22:29:34.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:34.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:29:34.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509447 2026-02-22T22:29:34.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509447 2026-02-22T22:29:34.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509447' 2026-02-22T22:29:34.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:34.965 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:29:35.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=81604378629 2026-02-22T22:29:35.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 81604378629 2026-02-22T22:29:35.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509447 3-81604378629' 2026-02-22T22:29:35.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:35.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836491 2026-02-22T22:29:35.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:35.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:29:35.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836491 2026-02-22T22:29:35.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:35.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836491 2026-02-22T22:29:35.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T22:29:35.030 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836491 2026-02-22T22:29:35.031 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:29:35.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-22T22:29:35.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:29:36.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:29:36.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:29:36.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T22:29:36.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:36.408 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672969 2026-02-22T22:29:36.408 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:36.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:29:36.409 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672969 2026-02-22T22:29:36.409 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:36.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672969 2026-02-22T22:29:36.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2026-02-22T22:29:36.410 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672969 2026-02-22T22:29:36.411 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:29:36.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672969 -lt 42949672969 2026-02-22T22:29:36.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:36.601 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509447 2026-02-22T22:29:36.602 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:36.602 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:29:36.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509447 2026-02-22T22:29:36.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:36.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509447 2026-02-22T22:29:36.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-22T22:29:36.604 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509447 2026-02-22T22:29:36.604 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:29:36.793 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509448 -lt 64424509447 2026-02-22T22:29:36.793 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:36.793 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-81604378629 2026-02-22T22:29:36.793 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:36.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:29:36.795 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-81604378629 2026-02-22T22:29:36.795 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:36.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=81604378629 2026-02-22T22:29:36.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 81604378629' 2026-02-22T22:29:36.796 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.3 seq 81604378629 2026-02-22T22:29:36.796 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:29:36.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 81604378630 -lt 81604378629 2026-02-22T22:29:36.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:29:36.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:36.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:37.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 == 0 2026-02-22T22:29:37.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:29:37.236 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:29:37.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:29:37.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:29:37.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:29:37.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:29:37.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:29:37.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=32 2026-02-22T22:29:37.419 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:29:37.419 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:37.419 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:37.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 32 = 32 2026-02-22T22:29:37.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:29:37.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:29:37.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-22T22:29:37.851 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:37.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2026-02-22T22:29:38.040 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:38.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2026-02-22T22:29:38.228 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:38.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2026-02-22T22:29:38.412 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:38.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-22T22:29:38.555 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:38.555 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-22T22:29:38.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-22T22:29:38.689 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:38.689 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-22T22:29:38.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-22T22:29:38.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2026-02-22T22:29:38.881 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:38.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2026-02-22T22:29:38.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2026-02-22T22:29:39.068 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:39.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2026-02-22T22:29:39.337 INFO:tasks.workunit.client.0.trial123.stderr:set pool 1 min_size to 2 2026-02-22T22:29:39.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2026-02-22T22:29:39.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:29:39.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:29:39.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:29:39.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:29:39.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:29:39.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:29:39.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:29:39.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:29:39.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:29:39.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:29:39.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:29:39.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:29:39.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:29:39.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:29:39.437 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:29:39.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:29:39.623 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:29:39.623 INFO:tasks.workunit.client.0.trial123.stderr:2 2026-02-22T22:29:39.623 INFO:tasks.workunit.client.0.trial123.stderr:3' 2026-02-22T22:29:39.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:29:39.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:39.623 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:29:39.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836494 2026-02-22T22:29:39.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836494 2026-02-22T22:29:39.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836494' 2026-02-22T22:29:39.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:39.685 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:29:39.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672972 2026-02-22T22:29:39.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672972 2026-02-22T22:29:39.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2026-02-22T22:29:39.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:39.746 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:29:39.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509450 2026-02-22T22:29:39.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509450 2026-02-22T22:29:39.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509450' 2026-02-22T22:29:39.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:39.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:29:39.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=81604378632 2026-02-22T22:29:39.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 81604378632 2026-02-22T22:29:39.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509450 3-81604378632' 2026-02-22T22:29:39.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:39.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836494 2026-02-22T22:29:39.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:39.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:29:39.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836494 2026-02-22T22:29:39.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:39.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836494 2026-02-22T22:29:39.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2026-02-22T22:29:39.869 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836494 2026-02-22T22:29:39.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:29:40.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836492 -lt 21474836494 2026-02-22T22:29:40.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:29:41.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:29:41.059 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:29:41.248 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836494 -lt 21474836494 2026-02-22T22:29:41.248 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:41.248 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672972 2026-02-22T22:29:41.248 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:41.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:29:41.249 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672972 2026-02-22T22:29:41.249 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:41.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672972 2026-02-22T22:29:41.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2026-02-22T22:29:41.251 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672972 2026-02-22T22:29:41.251 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:29:41.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672972 -lt 42949672972 2026-02-22T22:29:41.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:41.441 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509450 2026-02-22T22:29:41.442 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:41.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:29:41.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509450 2026-02-22T22:29:41.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:41.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509450 2026-02-22T22:29:41.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2026-02-22T22:29:41.444 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509450 2026-02-22T22:29:41.445 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:29:41.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509450 -lt 64424509450 2026-02-22T22:29:41.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:41.634 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-81604378632 2026-02-22T22:29:41.634 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:41.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:29:41.635 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-81604378632 2026-02-22T22:29:41.635 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:41.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=81604378632 2026-02-22T22:29:41.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 81604378632' 2026-02-22T22:29:41.637 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.3 seq 81604378632 2026-02-22T22:29:41.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:29:41.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 81604378632 -lt 81604378632 2026-02-22T22:29:41.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:29:41.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:41.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:42.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 == 0 2026-02-22T22:29:42.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:29:42.080 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:29:42.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:29:42.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:29:42.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:29:42.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:29:42.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:29:42.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=32 2026-02-22T22:29:42.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:29:42.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:42.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:42.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 32 = 32 2026-02-22T22:29:42.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:29:42.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:29:42.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-22T22:29:42.698 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:42.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-22T22:29:42.885 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:42.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-22T22:29:43.023 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:43.023 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-22T22:29:43.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-22T22:29:43.158 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:43.159 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-22T22:29:43.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-22T22:29:43.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2026-02-22T22:29:43.348 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:43.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2026-02-22T22:29:43.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2026-02-22T22:29:43.534 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:43.535 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2026-02-22T22:29:43.535 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2026-02-22T22:29:43.726 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:43.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2026-02-22T22:29:43.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2026-02-22T22:29:43.915 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:43.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2026-02-22T22:29:43.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:43.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:43.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:43.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:43.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:45.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:45.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2026-02-22T22:29:45.416 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 is already down. 2026-02-22T22:29:45.422 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2026-02-22T22:29:45.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: wait_for_peered: local cmd= 2026-02-22T22:29:45.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: wait_for_peered: local num_peered=-1 2026-02-22T22:29:45.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: wait_for_peered: local cur_peered 2026-02-22T22:29:45.423 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: wait_for_peered: get_timeout_delays 90 .1 2026-02-22T22:29:45.424 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:29:45.424 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:29:45.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:29:45.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:29:45.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:29:45.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:29:45.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: wait_for_peered: local -a delays 2026-02-22T22:29:45.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: wait_for_peered: local -i loop=0 2026-02-22T22:29:45.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: flush_pg_stats 2026-02-22T22:29:45.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:29:45.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:29:45.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:29:45.694 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:29:45.694 INFO:tasks.workunit.client.0.trial123.stderr:2 2026-02-22T22:29:45.694 INFO:tasks.workunit.client.0.trial123.stderr:3' 2026-02-22T22:29:45.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:29:45.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:45.694 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:29:45.740 INFO:tasks.workunit.client.0.trial123.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-22T22:29:45.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq= 2026-02-22T22:29:45.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z '' 2026-02-22T22:29:45.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: continue 2026-02-22T22:29:45.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:45.741 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:29:45.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672975 2026-02-22T22:29:45.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672975 2026-02-22T22:29:45.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 1-42949672975' 2026-02-22T22:29:45.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:45.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:29:45.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509453 2026-02-22T22:29:45.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509453 2026-02-22T22:29:45.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 1-42949672975 2-64424509453' 2026-02-22T22:29:45.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:29:45.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:29:45.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=81604378636 2026-02-22T22:29:45.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 81604378636 2026-02-22T22:29:45.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 1-42949672975 2-64424509453 3-81604378636' 2026-02-22T22:29:45.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:45.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672975 2026-02-22T22:29:45.929 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:45.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:29:45.930 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672975 2026-02-22T22:29:45.931 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672975 2026-02-22T22:29:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2026-02-22T22:29:45.931 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672975 2026-02-22T22:29:45.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:29:46.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672973 -lt 42949672975 2026-02-22T22:29:46.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:29:47.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:29:47.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:29:47.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672975 -lt 42949672975 2026-02-22T22:29:47.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:47.303 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509453 2026-02-22T22:29:47.303 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:47.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:29:47.304 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509453 2026-02-22T22:29:47.305 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:47.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509453 2026-02-22T22:29:47.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2026-02-22T22:29:47.306 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509453 2026-02-22T22:29:47.306 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:29:47.495 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509454 -lt 64424509453 2026-02-22T22:29:47.495 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:29:47.496 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-81604378636 2026-02-22T22:29:47.496 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:29:47.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:29:47.497 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-81604378636 2026-02-22T22:29:47.497 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:29:47.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=81604378636 2026-02-22T22:29:47.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 81604378636' 2026-02-22T22:29:47.498 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.3 seq 81604378636 2026-02-22T22:29:47.498 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:29:47.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 81604378636 -lt 81604378636 2026-02-22T22:29:47.688 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: get_num_pgs 2026-02-22T22:29:47.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:47.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:47.939 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: test 32 == 0 2026-02-22T22:29:47.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1751: wait_for_peered: true 2026-02-22T22:29:47.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1755: wait_for_peered: get_num_active_or_peered 2026-02-22T22:29:47.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2026-02-22T22:29:47.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-22T22:29:47.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-22T22:29:47.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-22T22:29:47.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-22T22:29:48.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1755: wait_for_peered: cur_peered=32 2026-02-22T22:29:48.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1756: wait_for_peered: get_num_pgs 2026-02-22T22:29:48.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:29:48.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:29:48.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1756: wait_for_peered: test 32 = 32 2026-02-22T22:29:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1756: wait_for_peered: break 2026-02-22T22:29:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1771: wait_for_peered: return 0 2026-02-22T22:29:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-22T22:29:48.559 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2026-02-22T22:29:48.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-22T22:29:48.746 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:48.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-22T22:29:48.887 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:48.888 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-22T22:29:48.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-22T22:29:49.022 INFO:tasks.workunit.client.0.trial123.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-22T22:29:49.023 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-22T22:29:49.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:49.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:49.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:49.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:49.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:49.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:49.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:49.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:49.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:49.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:29:49.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:29:49.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:29:49.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:29:49.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:29:49.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:29:49.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:29:49.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:29:49.145 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:29:49.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:29:49.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:29:49.161 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:29:49.161 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.162 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:29:49.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:29:49.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:49.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:49.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:49.165 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:29:49.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:29:49.166 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:29:49.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:29:49.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:29:49.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:29:49.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:29:49.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:29:49.169 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:29:49.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:29:49.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:29:49.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:29:49.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:29:49.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:29:49.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:29:49.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2026-02-22T22:29:49.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:29:49.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62918 2026-02-22T22:29:49.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:29:49.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:29:49.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:29:49.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2026-02-22T22:29:49.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-22T22:29:49.196 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:29:49.196 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:49.196 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:49.196 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:49.196 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.196 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.197 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:49.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-22T22:29:49.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:29:49.224 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:49.225 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.225 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.225 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:29:49.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:49.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get fsid 2026-02-22T22:29:49.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:29:49.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:49.269 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:29:49.269 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:29:49.269 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:29:49.269 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:29:49.269 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:29:49.269 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:29:49.270 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:49.270 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.270 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.270 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:29:49.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:49.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get mon_host 2026-02-22T22:29:49.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2026-02-22T22:29:49.404 INFO:tasks.workunit.client.0.trial123.stderr:invalid mon(s) specified: [dne] 2026-02-22T22:29:49.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2026-02-22T22:29:49.482 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2026-02-22T22:29:49.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2026-02-22T22:29:49.559 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-22T22:29:49.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2026-02-22T22:29:49.637 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-22T22:29:49.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2026-02-22T22:29:49.732 INFO:tasks.workunit.client.0.trial123.stderr:mon.dne does not exist 2026-02-22T22:29:49.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:49.738 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:49.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:49.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:49.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:49.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:49.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:49.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-22T22:29:49.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:29:49.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:29:49.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:49.846 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:49.846 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:49.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:49.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:49.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:49.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:49.851 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:29:49.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:29:49.851 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:29:49.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:29:49.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:29:49.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:29:49.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:29:49.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:29:49.854 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:29:49.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:29:49.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:29:49.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:29:49.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:29:49.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:29:49.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:29:49.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-22T22:29:49.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-22T22:29:49.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2026-02-22T22:29:49.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:49.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:49.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:49.861 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:29:49.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:29:49.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:29:49.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:29:49.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:29:49.863 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:29:49.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:29:49.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:29:49.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:49.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:29:49.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:29:49.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:29:49.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:29:49.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:29:49.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:29:49.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:29:49.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2026-02-22T22:29:49.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:29:49.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62918 2026-02-22T22:29:49.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:29:49.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:29:49.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:29:49.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:29:49.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2026-02-22T22:29:49.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-22T22:29:49.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:29:49.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:49.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:49.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:49.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:49.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-22T22:29:49.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:29:49.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:29:49.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:49.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:29:49.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:29:49.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:29:49.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:29:49.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:29:49.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:29:49.919 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:49.919 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.919 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.919 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:29:49.920 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:49.920 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get fsid 2026-02-22T22:29:49.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:29:49.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:49.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:29:49.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:49.965 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:29:49.966 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:49.966 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get mon_host 2026-02-22T22:29:50.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-22T22:29:50.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:29:50.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:29:50.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T22:29:50.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:29:50.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2026-02-22T22:29:50.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-22T22:29:50.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:29:50.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:50.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:50.030 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:50.030 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:50.030 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:50.030 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:50.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-22T22:29:50.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:29:50.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T22:29:50.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:50.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:29:50.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:29:50.058 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:29:50.058 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:29:50.058 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:29:50.058 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:29:50.058 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:50.058 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:50.058 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:50.059 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:29:50.059 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:50.059 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get fsid 2026-02-22T22:29:50.105 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T22:29:50.105 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:50.105 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:29:50.105 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:50.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get mon_host 2026-02-22T22:29:50.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2026-02-22T22:29:50.244 INFO:tasks.workunit.client.0.trial123.stderr:invalid mon(s) specified: [dne] 2026-02-22T22:29:50.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:29:50.326 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2026-02-22T22:29:50.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:29:50.403 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2026-02-22T22:29:50.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2026-02-22T22:29:50.483 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2026-02-22T22:29:50.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:29:50.578 INFO:tasks.workunit.client.0.trial123.stderr:adding another mon that is not yet online will not break quorum 2026-02-22T22:29:50.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:29:50.680 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.a 2026-02-22T22:29:50.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:29:50.779 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.b 2026-02-22T22:29:50.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2026-02-22T22:29:50.880 INFO:tasks.workunit.client.0.trial123.stderr:mon.dne does not exist 2026-02-22T22:29:50.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:50.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:50.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:50.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:50.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:50.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:50.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:50.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:50.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:51.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:51.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:29:51.001 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:29:51.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:29:51.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:29:51.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:29:51.003 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:29:51.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:51.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:29:51.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:29:51.004 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:51.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:29:51.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:29:51.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:29:51.008 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:29:51.008 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.008 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:29:51.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:29:51.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:51.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:51.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:51.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:51.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:51.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:51.012 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:29:51.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:29:51.012 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:29:51.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:29:51.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:29:51.013 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:29:51.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:51.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:29:51.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:29:51.015 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:29:51.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:29:51.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:29:51.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:29:51.017 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:29:51.017 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.018 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:29:51.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:29:51.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:29:51.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2026-02-22T22:29:51.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:29:51.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62918 2026-02-22T22:29:51.021 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:29:51.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:29:51.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:29:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:29:51.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2026-02-22T22:29:51.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-22T22:29:51.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:29:51.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:51.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:51.041 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:51.041 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.041 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:51.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-22T22:29:51.066 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:29:51.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:29:51.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:51.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:29:51.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:29:51.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:51.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get fsid 2026-02-22T22:29:51.114 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:51.115 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.116 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.116 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:29:51.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:51.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get mon_host 2026-02-22T22:29:51.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-22T22:29:51.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:29:51.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:29:51.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T22:29:51.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:29:51.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2026-02-22T22:29:51.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-22T22:29:51.178 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:29:51.179 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:51.179 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:51.179 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:51.179 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.179 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.179 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:51.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-22T22:29:51.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:29:51.205 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T22:29:51.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:51.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:29:51.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:29:51.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:29:51.206 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:29:51.206 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:29:51.206 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:29:51.207 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:51.207 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.207 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.207 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:29:51.208 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:51.208 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get fsid 2026-02-22T22:29:51.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T22:29:51.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:51.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:29:51.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:29:51.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:29:51.253 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:51.254 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get mon_host 2026-02-22T22:29:51.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-22T22:29:51.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:29:51.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:29:51.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2026-02-22T22:29:51.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:29:51.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2026-02-22T22:29:51.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-22T22:29:51.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:29:51.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:29:51.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:29:51.317 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:29:51.317 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.317 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.317 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:29:51.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-22T22:29:51.342 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:29:51.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2026-02-22T22:29:51.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:51.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2026-02-22T22:29:51.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:29:51.344 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:29:51.344 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2026-02-22T22:29:51.344 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2026-02-22T22:29:51.344 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2026-02-22T22:29:51.344 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:51.344 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.344 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.345 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.c.asok 2026-02-22T22:29:51.345 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:51.345 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.c.asok config get fsid 2026-02-22T22:29:51.391 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2026-02-22T22:29:51.391 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:29:51.391 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2026-02-22T22:29:51.391 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:29:51.392 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.c.asok 2026-02-22T22:29:51.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:29:51.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.c.asok config get mon_host 2026-02-22T22:29:51.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2026-02-22T22:29:51.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2026-02-22T22:29:51.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2026-02-22T22:29:51.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2026-02-22T22:29:51.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2026-02-22T22:29:51.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2026-02-22T22:29:51.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2026-02-22T22:29:51.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1771799451 2026-02-22T22:29:51.437 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:29:51.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799391 -lt 1771799451 ]] 2026-02-22T22:29:51.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:29:51.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:29:52.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:29:52.143 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:52.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:29:52.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:52.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2026-02-22T22:29:52.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-22T22:29:52.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2026-02-22T22:29:52.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2026-02-22T22:29:52.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2026-02-22T22:29:52.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2026-02-22T22:29:52.239 INFO:tasks.workunit.client.0.trial123.stderr:invalid mon(s) specified: [dne] 2026-02-22T22:29:52.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:29:52.338 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c) after stopping [a] 2026-02-22T22:29:52.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:29:52.437 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-22T22:29:52.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-22T22:29:52.537 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,b) after stopping [c] 2026-02-22T22:29:52.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2026-02-22T22:29:52.620 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2026-02-22T22:29:52.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2026-02-22T22:29:52.699 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2026-02-22T22:29:52.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2026-02-22T22:29:52.778 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2026-02-22T22:29:52.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:29:52.872 INFO:tasks.workunit.client.0.trial123.stderr:adding another mon that is not yet online will not break quorum 2026-02-22T22:29:52.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:29:52.971 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.a 2026-02-22T22:29:52.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:29:53.070 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.b 2026-02-22T22:29:53.076 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-22T22:29:53.171 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.c 2026-02-22T22:29:53.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2026-02-22T22:29:53.178 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:29:53.178 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:29:53.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:29:53.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:29:53.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2026-02-22T22:29:53.283 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2026-02-22T22:29:53.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1771799453 2026-02-22T22:29:53.284 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:29:53.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799393 -lt 1771799453 ]] 2026-02-22T22:29:53.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:29:53.286 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:29:56.385 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:29:56.385 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:56.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:29:56.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:56.387 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:29:56.387 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:29:56.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:29:56.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799396 -lt 1771799453 ]] 2026-02-22T22:29:56.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:29:56.389 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:29:59.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:29:59.489 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:59.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:59.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:29:59.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:29:59.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:29:59.492 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:29:59.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799399 -lt 1771799453 ]] 2026-02-22T22:29:59.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:29:59.494 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:29:59.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:29:59.593 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:59.594 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:59.594 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:29:59.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:29:59.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:29:59.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:29:59.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799399 -lt 1771799453 ]] 2026-02-22T22:29:59.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:29:59.597 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:29:59.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:29:59.698 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:59.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:29:59.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:29:59.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:29:59.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:29:59.701 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:29:59.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799399 -lt 1771799453 ]] 2026-02-22T22:29:59.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:29:59.703 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:02.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:02.803 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:02.804 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:02.804 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:02.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:02.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:02.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:02.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799402 -lt 1771799453 ]] 2026-02-22T22:30:02.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:02.807 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:02.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:02.907 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:02.908 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:02.908 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:02.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:02.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:02.910 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:02.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799402 -lt 1771799453 ]] 2026-02-22T22:30:02.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:02.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.010 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.013 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.014 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.112 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.113 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.113 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.114 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.215 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.215 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.215 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.320 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.320 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.321 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.322 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.323 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.422 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.423 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.423 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.423 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.425 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.426 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.426 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.524 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.527 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.528 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.628 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:03.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:03.730 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:03.731 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:03.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:03.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:03.733 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:03.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799403 -lt 1771799453 ]] 2026-02-22T22:30:03.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-22T22:30:03.734 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:15.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:15.835 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:15.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2026-02-22T22:30:15.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:29:51.033366Z","created":"2026-02-22T22:29:51.033366Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:15.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2026-02-22T22:30:15.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-22T22:30:15.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2026-02-22T22:30:15.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2026-02-22T22:30:15.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2026-02-22T22:30:15.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:30:15.914 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2026-02-22T22:30:15.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:30:16.011 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-22T22:30:16.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-22T22:30:16.094 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2026-02-22T22:30:16.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:30:16.171 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-22T22:30:16.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:30:16.247 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-22T22:30:16.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:30:16.344 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.b 2026-02-22T22:30:16.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-22T22:30:16.427 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-22T22:30:16.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-22T22:30:16.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:30:16.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:30:16.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:30:16.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:30:16.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:30:16.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:30:16.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-22T22:30:16.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:30:16.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:30:16.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:30:16.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:30:16.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:30:16.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:30:16.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:30:16.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:30:16.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:30:16.544 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:30:16.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:30:16.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:30:16.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:30:16.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:30:16.546 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:30:16.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:16.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:30:16.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:30:16.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:16.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:30:16.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:30:16.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:30:16.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:30:16.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:30:16.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:30:16.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:30:16.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-22T22:30:16.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-22T22:30:16.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2026-02-22T22:30:16.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2026-02-22T22:30:16.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:30:16.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:30:16.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:30:16.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:30:16.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:30:16.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:30:16.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:30:16.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:30:16.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:30:16.557 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:30:16.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:30:16.557 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:30:16.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:30:16.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:30:16.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:30:16.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:16.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:30:16.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:30:16.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:16.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:30:16.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:30:16.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:30:16.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:30:16.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:30:16.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:30:16.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:30:16.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2026-02-22T22:30:16.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:30:16.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62918 2026-02-22T22:30:16.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:30:16.567 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2026-02-22T22:30:16.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:16.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-22T22:30:16.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:16.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:30:16.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:16.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:30:16.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:16.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:16.613 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:30:16.613 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:30:16.613 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:30:16.614 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:16.614 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.615 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.615 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:30:16.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:16.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get fsid 2026-02-22T22:30:16.660 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:30:16.660 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:16.660 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:30:16.660 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:16.660 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:16.660 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:30:16.661 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:30:16.661 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:30:16.661 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:16.661 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.661 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.661 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:30:16.661 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:16.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get mon_host 2026-02-22T22:30:16.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-22T22:30:16.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:16.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:16.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T22:30:16.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:16.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2026-02-22T22:30:16.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-22T22:30:16.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:16.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:16.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:16.723 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:16.723 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.724 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.724 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:16.724 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-22T22:30:16.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:16.749 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:30:16.750 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:16.751 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.751 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.751 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:30:16.752 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:16.752 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get fsid 2026-02-22T22:30:16.798 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T22:30:16.798 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:16.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:30:16.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:16.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:16.799 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:30:16.799 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:30:16.799 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:30:16.799 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:16.800 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.800 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.800 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:30:16.800 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:16.800 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get mon_host 2026-02-22T22:30:16.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-22T22:30:16.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:16.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:16.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2026-02-22T22:30:16.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:16.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2026-02-22T22:30:16.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-22T22:30:16.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:16.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:16.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:16.862 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:16.862 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.862 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:16.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-22T22:30:16.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:16.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2026-02-22T22:30:16.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:16.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2026-02-22T22:30:16.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:16.890 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.c.asok 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:16.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.c.asok config get fsid 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2026-02-22T22:30:16.937 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2026-02-22T22:30:16.938 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:16.938 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:16.938 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:16.938 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.c.asok 2026-02-22T22:30:16.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:16.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.c.asok config get mon_host 2026-02-22T22:30:16.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-22T22:30:16.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:16.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:16.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2026-02-22T22:30:16.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:16.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2026-02-22T22:30:16.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-22T22:30:17.000 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:17.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:17.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:17.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:17.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:17.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:17.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:17.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-22T22:30:17.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:17.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2026-02-22T22:30:17.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:17.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2026-02-22T22:30:17.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:17.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:17.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2026-02-22T22:30:17.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2026-02-22T22:30:17.028 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2026-02-22T22:30:17.028 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:17.028 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:17.028 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:17.028 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.d.asok 2026-02-22T22:30:17.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:17.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.d.asok config get fsid 2026-02-22T22:30:17.073 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:17.074 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:17.075 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:17.075 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.d.asok 2026-02-22T22:30:17.075 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:17.075 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.d.asok config get mon_host 2026-02-22T22:30:17.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2026-02-22T22:30:17.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2026-02-22T22:30:17.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2026-02-22T22:30:17.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2026-02-22T22:30:17.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2026-02-22T22:30:17.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2026-02-22T22:30:17.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2026-02-22T22:30:17.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1771799477 2026-02-22T22:30:17.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:17.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799417 -lt 1771799477 ]] 2026-02-22T22:30:17.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:30:17.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:17.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:17.421 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:17.422 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:17.422 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:30:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2026-02-22T22:30:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-22T22:30:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2026-02-22T22:30:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2026-02-22T22:30:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2026-02-22T22:30:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2026-02-22T22:30:17.516 INFO:tasks.workunit.client.0.trial123.stderr:invalid mon(s) specified: [dne] 2026-02-22T22:30:17.521 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:30:17.614 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-22T22:30:17.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:30:17.711 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,c,d) after stopping [b] 2026-02-22T22:30:17.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-22T22:30:17.809 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,b,d) after stopping [c] 2026-02-22T22:30:17.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-22T22:30:17.906 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,b,c) after stopping [d] 2026-02-22T22:30:17.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2026-02-22T22:30:17.985 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2026-02-22T22:30:17.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2026-02-22T22:30:18.060 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2026-02-22T22:30:18.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:30:21.156 INFO:tasks.workunit.client.0.trial123.stderr:adding another mon that is not yet online will not break quorum 2026-02-22T22:30:21.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:30:21.255 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.a 2026-02-22T22:30:21.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:30:21.351 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.b 2026-02-22T22:30:21.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-22T22:30:21.450 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.c 2026-02-22T22:30:21.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-22T22:30:21.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:30:21.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:30:21.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:30:21.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:30:21.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2026-02-22T22:30:21.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2026-02-22T22:30:21.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1771799481 2026-02-22T22:30:21.564 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:21.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799421 -lt 1771799481 ]] 2026-02-22T22:30:21.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:21.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:24.668 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:24.668 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:24.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:24.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:24.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:24.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799424 -lt 1771799481 ]] 2026-02-22T22:30:24.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:24.672 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:24.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:24.774 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:24.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:24.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:24.777 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:24.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799424 -lt 1771799481 ]] 2026-02-22T22:30:24.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:24.778 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:24.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:24.879 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:24.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:24.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:24.882 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:24.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799424 -lt 1771799481 ]] 2026-02-22T22:30:24.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:24.883 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:24.983 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:24.983 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.984 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:24.984 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:24.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:24.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:24.986 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:24.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799424 -lt 1771799481 ]] 2026-02-22T22:30:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:24.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:25.087 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:25.087 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:25.088 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:25.088 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:25.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:25.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:25.090 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:25.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799425 -lt 1771799481 ]] 2026-02-22T22:30:25.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:25.091 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:28.192 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:28.192 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.192 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.193 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:28.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:28.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:28.195 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:28.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799428 -lt 1771799481 ]] 2026-02-22T22:30:28.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:28.196 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:28.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:28.298 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:28.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:28.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:28.301 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:28.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799428 -lt 1771799481 ]] 2026-02-22T22:30:28.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:28.302 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:28.402 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:28.403 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.403 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.403 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:28.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:28.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:28.406 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:28.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799428 -lt 1771799481 ]] 2026-02-22T22:30:28.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:28.407 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:28.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:28.508 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.509 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:28.509 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:28.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:30:28.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:30:28.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:28.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799428 -lt 1771799481 ]] 2026-02-22T22:30:28.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:30:28.512 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:37.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:37.616 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:37.616 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:16.578680Z","created":"2026-02-22T22:30:16.578680Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:37.616 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:30:37.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2026-02-22T22:30:37.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-22T22:30:37.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2026-02-22T22:30:37.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2026-02-22T22:30:37.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2026-02-22T22:30:37.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:30:37.710 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-22T22:30:37.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:30:37.789 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-22T22:30:37.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-22T22:30:37.864 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-22T22:30:37.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-22T22:30:37.941 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-22T22:30:37.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:30:41.036 INFO:tasks.workunit.client.0.trial123.stderr:adding another mon that is not yet online will not break quorum 2026-02-22T22:30:41.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:30:41.133 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.a 2026-02-22T22:30:41.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:30:41.231 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.b 2026-02-22T22:30:41.238 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-22T22:30:41.332 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.c 2026-02-22T22:30:41.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2026-02-22T22:30:41.431 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.d 2026-02-22T22:30:41.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-22T22:30:41.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:30:41.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:30:41.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:30:41.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:30:41.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:30:41.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:30:41.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-22T22:30:41.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:30:41.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:30:41.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:30:41.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:30:41.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:30:41.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:30:41.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:30:41.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:30:41.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:30:41.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:30:41.551 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:30:41.552 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:30:41.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:30:41.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:30:41.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:30:41.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:41.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:30:41.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:30:41.554 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:41.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:30:41.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:30:41.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:30:41.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:30:41.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:30:41.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:30:41.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:30:41.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:30:41.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:30:41.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:30:41.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:30:41.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:30:41.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:30:41.566 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:30:41.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:30:41.566 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:30:41.567 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:30:41.567 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:30:41.568 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:30:41.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:41.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:30:41.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:30:41.569 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:30:41.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:30:41.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:30:41.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:30:41.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:30:41.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:30:41.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:30:41.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:30:41.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2026-02-22T22:30:41.575 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:30:41.575 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.575 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.576 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62918 2026-02-22T22:30:41.578 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=fbf76503-940a-4c92-81d7-655e5809850f --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:41.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:30:41.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:41.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2026-02-22T22:30:41.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-22T22:30:41.594 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:41.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:41.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:41.595 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:41.595 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.595 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:41.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-22T22:30:41.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:41.623 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:30:41.623 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:41.623 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:30:41.623 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:41.623 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:41.623 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:30:41.623 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:30:41.624 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:30:41.624 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:41.624 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.624 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.624 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:30:41.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:41.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get fsid 2026-02-22T22:30:41.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:30:41.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.670 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.a.asok 2026-02-22T22:30:41.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:41.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.a.asok config get mon_host 2026-02-22T22:30:41.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-22T22:30:41.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:41.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:41.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T22:30:41.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:41.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2026-02-22T22:30:41.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-22T22:30:41.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:41.733 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:41.733 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:41.733 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:41.733 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.733 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.733 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:41.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-22T22:30:41.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:41.760 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T22:30:41.760 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:41.760 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:30:41.760 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:41.760 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:41.760 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:41.761 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get fsid 2026-02-22T22:30:41.804 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T22:30:41.804 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:41.804 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T22:30:41.804 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.b.asok 2026-02-22T22:30:41.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:41.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.b.asok config get mon_host 2026-02-22T22:30:41.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-22T22:30:41.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:41.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:41.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2026-02-22T22:30:41.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:41.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2026-02-22T22:30:41.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-22T22:30:41.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:41.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:41.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:41.868 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:41.868 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.868 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:41.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-22T22:30:41.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:41.895 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2026-02-22T22:30:41.895 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.c.asok 2026-02-22T22:30:41.896 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:41.897 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.c.asok config get fsid 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2026-02-22T22:30:41.943 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:41.944 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:41.944 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:41.944 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.c.asok 2026-02-22T22:30:41.944 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:41.944 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.c.asok config get mon_host 2026-02-22T22:30:41.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-22T22:30:41.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:41.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:41.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2026-02-22T22:30:41.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:41.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2026-02-22T22:30:41.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:42.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-22T22:30:42.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:42.032 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2026-02-22T22:30:42.032 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:42.032 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.d.asok 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:42.033 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.d.asok config get fsid 2026-02-22T22:30:42.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2026-02-22T22:30:42.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:42.079 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:42.080 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.d.asok 2026-02-22T22:30:42.080 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:42.080 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.d.asok config get mon_host 2026-02-22T22:30:42.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2026-02-22T22:30:42.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2026-02-22T22:30:42.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:30:42.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2026-02-22T22:30:42.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:30:42.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2026-02-22T22:30:42.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2026-02-22T22:30:42.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:30:42.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:30:42.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:30:42.143 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:30:42.143 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:42.143 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:42.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62918/$cluster-$name.asok' 2026-02-22T22:30:42.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.62918/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2026-02-22T22:30:42.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:30:42.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2026-02-22T22:30:42.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:42.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2026-02-22T22:30:42.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:30:42.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:30:42.172 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2026-02-22T22:30:42.172 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2026-02-22T22:30:42.172 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2026-02-22T22:30:42.172 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:42.172 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:42.172 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:42.172 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.e.asok 2026-02-22T22:30:42.173 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:42.173 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.e.asok config get fsid 2026-02-22T22:30:42.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2026-02-22T22:30:42.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:30:42.217 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62918/ceph-mon.e.asok 2026-02-22T22:30:42.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:30:42.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62918/ceph-mon.e.asok config get mon_host 2026-02-22T22:30:42.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2026-02-22T22:30:42.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2026-02-22T22:30:42.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2026-02-22T22:30:42.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2026-02-22T22:30:42.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2026-02-22T22:30:42.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2026-02-22T22:30:42.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2026-02-22T22:30:42.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1771799502 2026-02-22T22:30:42.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:30:42.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799442 -lt 1771799502 ]] 2026-02-22T22:30:42.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2026-02-22T22:30:42.267 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:30:54.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:30:54.370 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:54.370 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:30:54.370 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2026-02-22T22:30:54.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2026-02-22T22:30:54.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-22T22:30:54.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2026-02-22T22:30:54.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2026-02-22T22:30:54.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2026-02-22T22:30:54.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2026-02-22T22:30:54.467 INFO:tasks.workunit.client.0.trial123.stderr:invalid mon(s) specified: [dne] 2026-02-22T22:30:54.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:30:54.564 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-22T22:30:54.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:30:54.664 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2026-02-22T22:30:54.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-22T22:30:54.761 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2026-02-22T22:30:54.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-22T22:30:54.859 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2026-02-22T22:30:54.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-22T22:30:54.958 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2026-02-22T22:30:54.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2026-02-22T22:30:58.058 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2026-02-22T22:30:58.064 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2026-02-22T22:31:01.157 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2026-02-22T22:31:01.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2026-02-22T22:31:01.238 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2026-02-22T22:31:01.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:31:01.333 INFO:tasks.workunit.client.0.trial123.stderr:adding another mon that is not yet online will not break quorum 2026-02-22T22:31:01.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:31:01.432 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.a 2026-02-22T22:31:01.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:31:01.530 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.b 2026-02-22T22:31:01.535 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-22T22:31:01.627 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.c 2026-02-22T22:31:01.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-22T22:31:01.724 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.d 2026-02-22T22:31:01.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-22T22:31:01.820 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.e 2026-02-22T22:31:01.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-22T22:31:01.826 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:31:01.827 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:31:01.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:31:01.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:31:01.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:31:01.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:31:01.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2026-02-22T22:31:01.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2026-02-22T22:31:01.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2026-02-22T22:31:01.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2026-02-22T22:31:01.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2026-02-22T22:31:01.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2026-02-22T22:31:01.934 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2026-02-22T22:31:01.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1771799521 2026-02-22T22:31:01.934 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:01.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799461 -lt 1771799521 ]] 2026-02-22T22:31:01.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:01.936 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:05.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:05.040 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:05.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:05.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:05.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:05.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:05.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:05.043 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799465 -lt 1771799521 ]] 2026-02-22T22:31:05.043 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:05.044 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:05.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:05.144 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:05.145 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:05.145 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:05.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:05.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:05.147 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:05.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799465 -lt 1771799521 ]] 2026-02-22T22:31:05.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:05.148 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.250 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.250 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.251 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.252 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.354 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.354 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.358 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.458 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.462 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.562 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.566 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.666 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.769 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.769 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.769 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.772 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.773 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.871 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.872 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.872 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.874 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.875 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:08.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:08.973 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.974 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:08.974 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:08.976 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:08.976 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:08.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:08.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799468 -lt 1771799521 ]] 2026-02-22T22:31:08.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-22T22:31:08.977 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:22.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:22.483 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:22.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2026-02-22T22:31:22.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:22.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2026-02-22T22:31:22.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-22T22:31:22.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2026-02-22T22:31:22.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2026-02-22T22:31:22.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2026-02-22T22:31:22.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:31:22.580 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-22T22:31:22.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:31:25.682 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (c,d,e) after stopping [b] 2026-02-22T22:31:25.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-22T22:31:28.784 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,d,e) after stopping [c] 2026-02-22T22:31:28.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-22T22:31:28.885 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,e) after stopping [d] 2026-02-22T22:31:28.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-22T22:31:28.984 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-22T22:31:28.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:31:29.083 INFO:tasks.workunit.client.0.trial123.stderr:adding another mon that is not yet online will not break quorum 2026-02-22T22:31:29.088 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:31:32.182 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.a 2026-02-22T22:31:32.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:31:32.284 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.b 2026-02-22T22:31:32.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-22T22:31:32.385 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.c 2026-02-22T22:31:32.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-22T22:31:32.484 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.d 2026-02-22T22:31:32.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-22T22:31:32.583 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.e 2026-02-22T22:31:32.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2026-02-22T22:31:32.590 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:31:32.590 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:31:32.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:31:32.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:31:32.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2026-02-22T22:31:32.696 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2026-02-22T22:31:32.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1771799552 2026-02-22T22:31:32.697 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:32.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799492 -lt 1771799552 ]] 2026-02-22T22:31:32.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:32.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:32.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:32.796 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:32.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:32.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:32.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:32.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:32.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:32.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799492 -lt 1771799552 ]] 2026-02-22T22:31:32.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:32.800 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:32.900 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:32.900 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:32.901 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:32.901 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:32.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:32.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:32.903 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:32.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799492 -lt 1771799552 ]] 2026-02-22T22:31:32.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:32.904 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:33.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:33.005 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:33.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:33.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:33.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:33.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:33.008 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:33.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799493 -lt 1771799552 ]] 2026-02-22T22:31:33.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:33.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:36.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:36.108 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:36.110 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:36.110 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:36.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:36.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:36.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:36.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799496 -lt 1771799552 ]] 2026-02-22T22:31:36.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:36.112 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:39.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:39.212 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:39.213 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:39.213 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:39.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:39.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:39.214 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:39.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799499 -lt 1771799552 ]] 2026-02-22T22:31:39.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:39.215 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:39.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:39.315 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:39.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:39.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:39.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:39.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:39.318 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:39.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799499 -lt 1771799552 ]] 2026-02-22T22:31:39.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:39.320 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:39.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:39.417 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:39.417 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:39.418 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:39.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:39.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:39.420 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:39.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799499 -lt 1771799552 ]] 2026-02-22T22:31:39.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:39.421 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:42.522 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:42.522 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:42.523 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:42.523 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:42.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2026-02-22T22:31:42.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-22T22:31:42.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2026-02-22T22:31:42.526 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1771799502 -lt 1771799552 ]] 2026-02-22T22:31:42.526 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-22T22:31:42.526 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-22T22:31:48.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2026-02-22T22:31:48.626 INFO:tasks.workunit.client.0.trial123.stderr:{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:48.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"fbf76503-940a-4c92-81d7-655e5809850f","modified":"2026-02-22T22:30:41.587978Z","created":"2026-02-22T22:30:41.587978Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-22T22:31:48.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2026-02-22T22:31:48.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2026-02-22T22:31:48.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-22T22:31:48.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2026-02-22T22:31:48.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2026-02-22T22:31:48.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2026-02-22T22:31:48.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-22T22:31:48.725 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-22T22:31:48.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-22T22:31:48.806 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-22T22:31:48.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-22T22:31:48.882 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-22T22:31:48.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-22T22:31:51.961 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-22T22:31:51.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-22T22:31:52.057 INFO:tasks.workunit.client.0.trial123.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-22T22:31:52.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-22T22:31:55.137 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-22T22:31:55.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-22T22:31:55.232 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.a 2026-02-22T22:31:55.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-22T22:31:55.311 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: removing mon.b would break quorum 2026-02-22T22:31:55.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-22T22:31:55.391 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-22T22:31:55.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-22T22:31:55.467 INFO:tasks.workunit.client.0.trial123.stderr:Error EBUSY: removing mon.d would break quorum 2026-02-22T22:31:55.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-22T22:31:58.564 INFO:tasks.workunit.client.0.trial123.stderr:safe to remove mon.e 2026-02-22T22:31:58.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-22T22:31:58.571 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:31:58.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:31:58.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:31:58.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:31:58.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:31:58.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:31:58.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-22T22:31:58.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:31:58.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:31:58.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:31:58.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:31:58.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:31:58.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:31:58.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:31:58.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:31:58.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:31:58.685 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:31:58.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:31:58.686 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:31:58.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:31:58.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:31:58.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:31:58.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:31:58.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:31:58.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:31:58.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:31:58.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:31:58.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:31:58.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:31:58.701 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:31:58.701 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:58.701 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:31:58.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:31:58.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:31:58.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:31:58.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T22:31:58.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/ok-to-stop 0 2026-02-22T22:31:58.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2026-02-22T22:31:58.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T22:31:58.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2026-02-22T22:31:58.706 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:31:58.706 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:31:58.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:31:58.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:31:58.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:31:58.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:31:58.707 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:31:58.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:31:58.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:31:58.712 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:31:58.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:31:58.713 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:31:58.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:31:58.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:31:58.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:31:58.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:31:58.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:31:58.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T22:31:58.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2026-02-22T22:31:58.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:31:58.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:58.715 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62918 2026-02-22T22:31:58.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62918 2026-02-22T22:31:58.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:31:58.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:31:58.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T22:31:58.716 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T22:31:58.716 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T22:31:58.779 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2026-02-22T22:31:58.779 DEBUG:teuthology.orchestra.run.trial123:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2026-02-22T22:31:58.843 INFO:tasks.workunit.client.0.trial123.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/rados-striper 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2026-02-22T22:31:58.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none ' 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:31:58.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2026-02-22T22:31:58.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:31:58.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:31:58.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:31:58.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:31:58.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:31:58.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:31:58.850 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:31:58.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:31:58.851 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:31:58.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:31:58.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:31:58.852 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:31:58.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:31:58.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:31:58.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:31:58.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:31:58.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:31:58.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:31:58.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2026-02-22T22:31:58.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:31:58.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:58.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:31:58.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.78769 2026-02-22T22:31:58.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:31:58.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:31:58.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2026-02-22T22:31:58.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:31:58.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:58.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:31:58.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.78769 2026-02-22T22:31:58.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:31:58.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:31:58.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:31:58.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2026-02-22T22:31:58.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2026-02-22T22:31:58.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:31:58.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:31:58.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:31:58.878 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:31:58.878 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:58.878 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:31:58.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.78769/$cluster-$name.asok' 2026-02-22T22:31:58.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:31:58.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:31:58.905 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:31:58.906 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:31:58.906 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:58.907 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:31:58.907 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.78769/ceph-mon.a.asok 2026-02-22T22:31:58.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:31:58.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.78769/ceph-mon.a.asok config get fsid 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:31:58.951 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:31:58.952 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:31:58.952 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:58.952 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:31:58.952 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.78769/ceph-mon.a.asok 2026-02-22T22:31:58.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:31:58.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.78769/ceph-mon.a.asok config get mon_host 2026-02-22T22:31:58.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2026-02-22T22:31:58.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2026-02-22T22:31:59.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.78769/$cluster-$name.asok' 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' 2026-02-22T22:31:59.001 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:31:59.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2026-02-22T22:31:59.003 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:31:59.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=723ec01a-a3da-429a-8512-3ee3866e6bd1 2026-02-22T22:31:59.003 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 723ec01a-a3da-429a-8512-3ee3866e6bd1 2026-02-22T22:31:59.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 723ec01a-a3da-429a-8512-3ee3866e6bd1' 2026-02-22T22:31:59.004 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:31:59.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfg5tp3+mNABAAfWu8vw6bOa4CVvZh37L+MQ== 2026-02-22T22:31:59.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfg5tp3+mNABAAfWu8vw6bOa4CVvZh37L+MQ=="}' 2026-02-22T22:31:59.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 723ec01a-a3da-429a-8512-3ee3866e6bd1 -i td/rados-striper/0/new.json 2026-02-22T22:31:59.111 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:31:59.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2026-02-22T22:31:59.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfg5tp3+mNABAAfWu8vw6bOa4CVvZh37L+MQ== --osd-uuid 723ec01a-a3da-429a-8512-3ee3866e6bd1 2026-02-22T22:31:59.134 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:31:59.132+0000 7fa35402f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:31:59.135 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:31:59.134+0000 7fa35402f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:31:59.136 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:31:59.135+0000 7fa35402f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:31:59.137 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:31:59.135+0000 7fa35402f780 -1 bdev(0x556012291c00 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:31:59.137 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:31:59.135+0000 7fa35402f780 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2026-02-22T22:32:01.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2026-02-22T22:32:01.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:32:01.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:32:01.264 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:32:01.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:32:01.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:32:01.364 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:32:01.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:32:01.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:32:01.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:32:01.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:32:01.380 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:01.378+0000 7f54a670c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:01.381 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:01.379+0000 7f54a670c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:01.382 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:01.380+0000 7f54a670c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:01.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:32:01.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:32:01.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:32:01.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:01.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:01.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:01.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:01.462 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:01.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:01.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:01.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:02.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:02.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:02.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:32:02.555 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:02.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:02.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:02.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:02.728 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:02.726+0000 7f54a670c780 -1 Falling back to public interface 2026-02-22T22:32:03.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:03.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:03.652 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:03.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:32:03.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:03.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:03.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:04.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:04.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:04.750 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:32:04.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:32:04.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:04.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:04.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:05.418 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:05.416+0000 7f54a670c780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:32:05.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:05.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:05.846 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:32:05.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:32:05.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:05.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:05.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:06.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:06.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:06.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:32:06.948 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:32:06.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:06.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:07.043 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3542616524,v1:127.0.0.1:6801/3542616524] [v2:127.0.0.1:6802/3542616524,v1:127.0.0.1:6803/3542616524] exists,up 723ec01a-a3da-429a-8512-3ee3866e6bd1 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2026-02-22T22:32:07.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:07.048 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.78769/$cluster-$name.asok' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:32:07.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:32:07.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:32:07.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:32:07.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2026-02-22T22:32:07.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:32:07.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce8d1b3b-feda-4cca-b854-e6196554f23c 2026-02-22T22:32:07.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ce8d1b3b-feda-4cca-b854-e6196554f23c' 2026-02-22T22:32:07.050 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 ce8d1b3b-feda-4cca-b854-e6196554f23c 2026-02-22T22:32:07.051 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:32:07.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDng5tpPeZPAxAACL5kru8K9fbizZivf+MpoQ== 2026-02-22T22:32:07.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDng5tpPeZPAxAACL5kru8K9fbizZivf+MpoQ=="}' 2026-02-22T22:32:07.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce8d1b3b-feda-4cca-b854-e6196554f23c -i td/rados-striper/1/new.json 2026-02-22T22:32:07.157 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:07.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2026-02-22T22:32:07.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDng5tpPeZPAxAACL5kru8K9fbizZivf+MpoQ== --osd-uuid ce8d1b3b-feda-4cca-b854-e6196554f23c 2026-02-22T22:32:07.179 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:07.177+0000 7f3edd20c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:07.181 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:07.178+0000 7f3edd20c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:07.181 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:07.179+0000 7f3edd20c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:07.182 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:07.179+0000 7f3edd20c780 -1 bdev(0x55eb6b06bc00 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:32:07.182 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:07.179+0000 7f3edd20c780 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2026-02-22T22:32:09.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2026-02-22T22:32:09.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:32:09.305 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:32:09.305 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:32:09.305 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:32:09.404 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:32:09.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:32:09.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:32:09.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:32:09.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:32:09.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:32:09.420 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:09.417+0000 7f0ccc60d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:09.421 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:09.418+0000 7f0ccc60d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:09.422 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:09.419+0000 7f0ccc60d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:32:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:32:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:32:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:09.502 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:09.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:09.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:32:09.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:10.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:10.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:10.595 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:10.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:32:10.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:10.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:32:10.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:10.755 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:10.752+0000 7f0ccc60d780 -1 Falling back to public interface 2026-02-22T22:32:11.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:11.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:11.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:32:11.690 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:11.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:11.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:32:11.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:12.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:12.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:12.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:32:12.786 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:32:12.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:12.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:32:12.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:13.459 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:13.456+0000 7f0ccc60d780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:32:13.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:13.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:13.880 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:32:13.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:32:13.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:13.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:32:13.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:14.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:14.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:14.979 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:32:14.979 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:32:14.979 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:14.979 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:32:15.077 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/478085450,v1:127.0.0.1:6809/478085450] [v2:127.0.0.1:6810/478085450,v1:127.0.0.1:6811/478085450] exists,up ce8d1b3b-feda-4cca-b854-e6196554f23c 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:32:15.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:32:15.082 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.78769/$cluster-$name.asok' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:32:15.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f5284a5-b90d-4d7d-b713-20d29e911c1e 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7f5284a5-b90d-4d7d-b713-20d29e911c1e' 2026-02-22T22:32:15.084 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 7f5284a5-b90d-4d7d-b713-20d29e911c1e 2026-02-22T22:32:15.085 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:32:15.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvg5tpjIVjBRAA6DRfNixGSzXmIqsBYk76mQ== 2026-02-22T22:32:15.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvg5tpjIVjBRAA6DRfNixGSzXmIqsBYk76mQ=="}' 2026-02-22T22:32:15.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f5284a5-b90d-4d7d-b713-20d29e911c1e -i td/rados-striper/2/new.json 2026-02-22T22:32:15.195 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:15.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2026-02-22T22:32:15.203 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvg5tpjIVjBRAA6DRfNixGSzXmIqsBYk76mQ== --osd-uuid 7f5284a5-b90d-4d7d-b713-20d29e911c1e 2026-02-22T22:32:15.219 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:15.216+0000 7f5c145b2780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:15.220 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:15.217+0000 7f5c145b2780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:15.221 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:15.218+0000 7f5c145b2780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:15.222 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:15.219+0000 7f5c145b2780 -1 bdev(0x55931893bc00 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:32:15.222 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:15.219+0000 7f5c145b2780 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2026-02-22T22:32:17.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2026-02-22T22:32:17.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:32:17.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:32:17.349 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:32:17.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:32:17.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:32:17.449 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:32:17.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2d245a7a-5ba6-4c65-acc4-0fe5626a1681 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.78769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:32:17.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:32:17.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:32:17.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:32:17.464 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:17.462+0000 7f6534efe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:17.465 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:17.463+0000 7f6534efe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:17.466 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:17.464+0000 7f6534efe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:17.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:32:17.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:32:17.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:32:17.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:17.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:17.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:17.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:17.545 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:17.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:17.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:32:17.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:18.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:18.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:18.639 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:18.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:32:18.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:32:18.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:18.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:18.802 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:18.800+0000 7f6534efe780 -1 Falling back to public interface 2026-02-22T22:32:19.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:19.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:19.734 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:19.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:32:19.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:19.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:32:19.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:20.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:20.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:20.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:32:20.833 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:32:20.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:20.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:32:20.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:21.502 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:21.500+0000 7f6534efe780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:32:21.926 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:21.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:21.927 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:32:21.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:32:21.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:21.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:32:22.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:23.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:23.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:23.026 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:32:23.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:32:23.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:23.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:32:23.123 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1289470446,v1:127.0.0.1:6817/1289470446] [v2:127.0.0.1:6818/1289470446,v1:127.0.0.1:6819/1289470446] exists,up 7f5284a5-b90d-4d7d-b713-20d29e911c1e 2026-02-22T22:32:23.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:23.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:23.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:23.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2026-02-22T22:32:23.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:32:23.216 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:32:23.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:32:23.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:32:23.372 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:32:23.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:32:24.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:32:24.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2026-02-22T22:32:24.661 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records in 2026-02-22T22:32:24.661 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records out 2026-02-22T22:32:24.661 INFO:tasks.workunit.client.0.trial123.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 3.9172e-05 s, 31.5 MB/s 2026-02-22T22:32:24.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2026-02-22T22:32:24.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2026-02-22T22:32:24.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2026-02-22T22:32:24.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-22T22:32:24.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2026-02-22T22:32:24.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2026-02-22T22:32:24.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2026-02-22T22:32:24.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2026-02-22T22:32:24.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2026-02-22T22:32:24.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2026-02-22T22:32:24.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2026-02-22T22:32:24.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2026-02-22T22:32:24.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2026-02-22T22:32:24.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2026-02-22T22:32:24.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2026-02-22T22:32:24.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2026-02-22T22:32:24.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2026-02-22T22:32:24.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-22T22:32:24.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2026-02-22T22:32:24.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2026-02-22T22:32:24.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2026-02-22T22:32:24.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2026-02-22T22:32:24.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2026-02-22T22:32:24.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-22T22:32:24.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-22T22:32:24.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2026-02-22T22:32:24.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2026-02-22T22:32:24.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2026-02-22T22:32:24.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2026-02-22T22:32:24.900 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2026-02-22T22:32:24.900 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2026-02-22T22:32:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2026-02-22T22:32:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2026-02-22T22:32:24.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2026-02-22T22:32:24.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2026-02-22T22:32:24.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2026-02-22T22:32:24.936 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: local dir=td/rados-striper 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: expect_failure: shift 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: expect_failure: local 'expected=No data available' 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: shift 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: local success 2026-02-22T22:32:24.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-22T22:32:24.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: expect_failure: success=false 2026-02-22T22:32:24.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: false 2026-02-22T22:32:24.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: expect_failure: return 0 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: local dir=td/rados-striper 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: expect_failure: shift 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: expect_failure: local 'expected=No data available' 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: shift 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: local success 2026-02-22T22:32:24.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-22T22:32:24.968 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: expect_failure: success=false 2026-02-22T22:32:24.968 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: false 2026-02-22T22:32:24.968 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-22T22:32:24.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: expect_failure: return 0 2026-02-22T22:32:24.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2026-02-22T22:32:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2026-02-22T22:32:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: local dir=td/rados-striper 2026-02-22T22:32:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: expect_failure: shift 2026-02-22T22:32:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: expect_failure: local 'expected=No such file or directory' 2026-02-22T22:32:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: shift 2026-02-22T22:32:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: local success 2026-02-22T22:32:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: expect_failure: rados --pool rbd --striper stat toyfile 2026-02-22T22:32:25.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: expect_failure: success=false 2026-02-22T22:32:25.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: false 2026-02-22T22:32:25.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-22T22:32:25.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: expect_failure: return 0 2026-02-22T22:32:25.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2026-02-22T22:32:25.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: local dir=td/rados-striper 2026-02-22T22:32:25.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: expect_failure: shift 2026-02-22T22:32:25.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: expect_failure: local 'expected=No such file or directory' 2026-02-22T22:32:25.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: shift 2026-02-22T22:32:25.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: local success 2026-02-22T22:32:25.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2026-02-22T22:32:25.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: expect_failure: success=false 2026-02-22T22:32:25.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: false 2026-02-22T22:32:25.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: expect_failure: return 0 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:25.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:25.130 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:32:25.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:32:25.131 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:32:25.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:32:25.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:32:25.132 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:32:25.133 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:25.133 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:32:25.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:32:25.134 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:25.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:32:25.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:32:25.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2026-02-22T22:32:25.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:32:25.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:32:25.142 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.78769 2026-02-22T22:32:25.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:32:25.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:32:25.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/rados-striper 0 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:25.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:25.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:25.145 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:32:25.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:32:25.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:32:25.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:32:25.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:32:25.147 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:32:25.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:25.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:32:25.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:32:25.148 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:25.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:32:25.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T22:32:25.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2026-02-22T22:32:25.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:32:25.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.78769 2026-02-22T22:32:25.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.78769 2026-02-22T22:32:25.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:32:25.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:32:25.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T22:32:25.153 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T22:32:25.154 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T22:32:25.220 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2026-02-22T22:32:25.220 DEBUG:teuthology.orchestra.run.trial123:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2026-02-22T22:32:25.285 INFO:tasks.workunit.client.0.trial123.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2391: run_tests: export PATH=./bin:.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2391: run_tests: PATH=./bin:.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: run_tests: export CEPH_MON=127.0.0.1:7109 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: run_tests: CEPH_MON=127.0.0.1:7109 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: run_tests: export CEPH_ARGS 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: run_tests: uuidgen 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: run_tests: CEPH_ARGS+=' --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none ' 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: run_tests: export CEPH_CONF=/dev/null 2026-02-22T22:32:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: run_tests: CEPH_CONF=/dev/null 2026-02-22T22:32:25.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: run_tests: set 2026-02-22T22:32:25.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: run_tests: local 'funcs=test_activate_osd 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_activate_osd_after_mark_down 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_activate_osd_skip_benchmark 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_destroy_osd 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_display_logs 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_erasure_code_plugin_exists 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_expect_failure 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_flush_pg_stats 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_get_config 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_get_is_making_recovery_progress 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_get_last_scrub_stamp 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_get_not_primary 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_get_num_active_clean 2026-02-22T22:32:25.290 INFO:tasks.workunit.client.0.trial123.stderr:test_get_num_active_or_peered 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_get_num_pgs 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_get_op_scheduler 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_get_osd_id_used_by_pgs 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_get_osds 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_get_pg 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_get_primary 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_get_timeout_delays 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_is_clean 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_kill_daemon 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_kill_daemons 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_objectstore_tool 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_pg_schedule_scrub 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_pg_scrub 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_repair 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_run_in_background 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_run_mon 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_run_osd 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_set_config 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_setup 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_teardown 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_wait_background 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_wait_for_clean 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_wait_for_health_ok 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_wait_for_osd 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_wait_for_peered 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_wait_for_scrub 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:test_wait_osd_id_used_by_pgs' 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: run_tests: local dir=td/ceph-helpers 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_activate_osd td/ceph-helpers 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:32:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:32:25.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:25.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:25.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:25.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:25.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:25.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:25.293 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:32:25.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:32:25.294 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:32:25.295 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:32:25.295 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:32:25.295 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:32:25.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:25.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:32:25.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:32:25.296 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:25.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:32:25.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:32:25.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:32:25.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:32:25.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:25.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:32:25.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:32:25.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:32:25.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:32:25.301 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:32:25.301 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.301 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:25.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:32:25.302 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:32:25.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T22:32:25.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T22:32:25.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:32:25.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:32:25.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2026-02-22T22:32:25.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:32:25.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:32:25.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:32:25.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:32:25.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:32:25.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:32:25.323 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:32:25.323 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:25.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:25.324 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:25.324 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.324 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:25.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:25.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:32:25.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:32:25.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:32:25.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:32:25.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:32:25.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:32:25.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:32:25.352 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:32:25.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:32:25.395 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.396 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:25.397 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:32:25.397 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:32:25.397 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:32:25.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2026-02-22T22:32:25.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:32:25.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:32:25.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:32:25.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:32:25.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:32:25.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:32:25.538 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:32:25.538 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:25.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:25.539 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:25.539 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.539 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:25.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:25.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:32:25.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:32:25.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2026-02-22T22:32:25.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:32:25.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:32:25.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:32:25.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:32:25.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=60fe4a9f-e6d7-4c83-b589-400078f88b07 2026-02-22T22:32:25.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 60fe4a9f-e6d7-4c83-b589-400078f88b07' 2026-02-22T22:32:25.563 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 60fe4a9f-e6d7-4c83-b589-400078f88b07 2026-02-22T22:32:25.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:32:25.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5g5tpDwrtIRAA7ZqC+w51xdI3TdQ6Z8R0Qg== 2026-02-22T22:32:25.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5g5tpDwrtIRAA7ZqC+w51xdI3TdQ6Z8R0Qg=="}' 2026-02-22T22:32:25.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 60fe4a9f-e6d7-4c83-b589-400078f88b07 -i td/ceph-helpers/0/new.json 2026-02-22T22:32:25.674 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:25.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:32:25.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5g5tpDwrtIRAA7ZqC+w51xdI3TdQ6Z8R0Qg== --osd-uuid 60fe4a9f-e6d7-4c83-b589-400078f88b07 2026-02-22T22:32:25.698 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:25.697+0000 7f50fc716780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:25.700 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:25.698+0000 7f50fc716780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:25.700 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:25.699+0000 7f50fc716780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:25.701 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:25.699+0000 7f50fc716780 -1 bdev(0x55848016bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:32:25.701 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:25.699+0000 7f50fc716780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:32:28.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:32:28.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:32:28.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:32:28.595 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:32:28.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:32:28.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:32:28.852 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:32:28.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:32:28.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:32:28.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:32:28.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:32:28.867 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:28.865+0000 7f809a011780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:28.868 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:28.867+0000 7f809a011780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:28.869 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:28.868+0000 7f809a011780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:29.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:32:29.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:32:29.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:32:29.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:29.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:29.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:29.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:29.041 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:29.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:29.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:29.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:30.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:30.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:30.229 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:30.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:32:30.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:30.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:30.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:30.466 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:30.465+0000 7f809a011780 -1 Falling back to public interface 2026-02-22T22:32:31.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:31.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:31.419 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:31.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:32:31.420 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:31.420 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:31.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:32.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:32.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:32.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:32:32.607 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:32:32.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:32.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:32.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:33.398 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:33.397+0000 7f809a011780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:32:33.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:33.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:33.798 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:32:33.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:32:33.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:33.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:33.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:34.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:34.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:34.990 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:32:34.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:32:34.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:34.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:35.183 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3369307509,v1:127.0.0.1:6803/3369307509] [v2:127.0.0.1:6804/3369307509,v1:127.0.0.1:6805/3369307509] exists,up 60fe4a9f-e6d7-4c83-b589-400078f88b07 2026-02-22T22:32:35.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:35.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:35.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:35.184 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2026-02-22T22:32:35.184 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:32:35.184 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:32:35.185 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:35.185 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:35.185 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:35.185 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:32:35.185 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2026-02-22T22:32:35.185 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:32:35.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2026-02-22T22:32:35.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2026-02-22T22:32:35.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2026-02-22T22:32:35.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2026-02-22T22:32:35.233 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:35.233 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:35.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:35.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:35.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:36.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:36.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-22T22:32:36.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:32:36.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:36.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-22T22:32:36.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:32:36.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T22:32:36.547 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:32:36.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-22T22:32:36.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-22T22:32:36.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T22:32:36.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:32:36.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:32:36.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:32:36.560 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:36.558+0000 7f3618616780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:36.561 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:36.559+0000 7f3618616780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:36.562 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:36.560+0000 7f3618616780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:36.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T22:32:36.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:32:36.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:32:36.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:36.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:36.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:36.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:36.735 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:36.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:36.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:36.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:37.898 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:37.896+0000 7f3618616780 -1 Falling back to public interface 2026-02-22T22:32:37.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:37.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:37.925 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:37.926 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:32:37.926 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:37.926 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:38.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:39.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:39.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:39.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:32:39.114 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:39.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:39.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:39.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:40.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:40.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:40.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:32:40.303 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:32:40.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:40.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:40.491 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:40.854 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:40.852+0000 7f3618616780 -1 osd.0 5 log_to_monitors true 2026-02-22T22:32:41.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:41.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:41.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:32:41.492 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:32:41.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:41.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:41.682 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/3493697653,v1:127.0.0.1:6803/3493697653] [v2:127.0.0.1:6804/3493697653,v1:127.0.0.1:6805/3493697653] exists,up 60fe4a9f-e6d7-4c83-b589-400078f88b07 2026-02-22T22:32:41.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:41.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:41.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:41.683 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2026-02-22T22:32:41.683 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:32:41.683 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:32:41.683 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:41.683 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:41.683 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:41.684 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:32:41.684 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2026-02-22T22:32:41.684 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:32:41.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:41.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:41.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:41.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:41.840 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:32:41.841 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:32:41.841 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:32:41.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:32:41.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:32:41.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:32:41.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:41.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:32:41.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:32:41.845 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:41.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:32:41.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:32:41.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:32:41.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:32:41.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:41.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:41.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:32:41.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:32:41.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:32:41.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:41.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:41.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:41.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:41.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:41.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:41.854 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:32:41.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:32:41.854 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:32:41.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:32:41.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:32:41.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:32:41.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:41.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:32:41.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:32:41.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:41.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:32:41.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:32:41.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:32:41.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:32:41.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:41.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:41.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:32:41.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:32:41.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:32:41.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:32:41.860 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:32:41.860 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:41.860 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:41.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:32:41.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:32:41.882 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:32:41.882 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:41.882 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:41.882 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:41.882 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:41.882 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:41.883 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:41.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:32:41.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:41.911 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:41.912 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:41.912 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:32:41.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:32:41.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:32:41.960 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:32:41.960 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:32:41.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:32:42.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2026-02-22T22:32:42.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:32:42.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:32:42.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:32:42.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:32:42.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:32:42.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:32:42.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:32:42.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:42.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:42.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:42.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:42.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:42.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:42.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:32:42.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:32:42.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:42.129 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:32:42.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:32:42.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:32:42.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:32:42.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:32:42.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:32:42.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:32:42.132 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:32:42.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fba3efd-872b-4973-a110-aa43a417f30f 2026-02-22T22:32:42.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3fba3efd-872b-4973-a110-aa43a417f30f' 2026-02-22T22:32:42.132 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 3fba3efd-872b-4973-a110-aa43a417f30f 2026-02-22T22:32:42.133 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:32:42.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKhJtpenxFCBAA1sb7qXTxuIUs29Q0ngGtYg== 2026-02-22T22:32:42.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKhJtpenxFCBAA1sb7qXTxuIUs29Q0ngGtYg=="}' 2026-02-22T22:32:42.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fba3efd-872b-4973-a110-aa43a417f30f -i td/ceph-helpers/0/new.json 2026-02-22T22:32:42.244 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:42.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:32:42.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKhJtpenxFCBAA1sb7qXTxuIUs29Q0ngGtYg== --osd-uuid 3fba3efd-872b-4973-a110-aa43a417f30f 2026-02-22T22:32:42.270 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:42.268+0000 7f0692a22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:42.271 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:42.269+0000 7f0692a22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:42.272 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:42.270+0000 7f0692a22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:42.272 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:42.270+0000 7f0692a22780 -1 bdev(0x556c8cc91c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:32:42.272 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:42.270+0000 7f0692a22780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:32:45.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:32:45.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:32:45.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:32:45.166 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:32:45.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:32:45.423 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:32:45.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:32:45.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:32:45.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:32:45.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:32:45.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:32:45.440 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:45.438+0000 7fbfb9a95780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:45.442 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:45.439+0000 7fbfb9a95780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:45.443 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:45.440+0000 7fbfb9a95780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:45.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:46.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:46.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:46.802 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:46.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:32:46.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:46.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:46.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:47.026 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:47.023+0000 7fbfb9a95780 -1 Falling back to public interface 2026-02-22T22:32:47.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:47.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:47.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:32:47.990 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:47.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:47.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:48.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:49.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:49.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:49.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:32:49.179 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:32:49.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:49.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:49.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:49.721 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:49.718+0000 7fbfb9a95780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:32:50.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:50.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:50.367 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:32:50.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:32:50.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:50.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:50.559 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/899522609,v1:127.0.0.1:6803/899522609] [v2:127.0.0.1:6804/899522609,v1:127.0.0.1:6805/899522609] exists,up 3fba3efd-872b-4973-a110-aa43a417f30f 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:50.561 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:50.562 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:50.562 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:32:50.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-22T22:32:50.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:32:50.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2026-02-22T22:32:50.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2026-02-22T22:32:50.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2026-02-22T22:32:50.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2026-02-22T22:32:50.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:50.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:50.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:50.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:50.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:51.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:51.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2026-02-22T22:32:52.112 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 is already down. 2026-02-22T22:32:52.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2026-02-22T22:32:52.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2026-02-22T22:32:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:32:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:52.120 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:52.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:52.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2026-02-22T22:32:52.307 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/899522609,v1:127.0.0.1:6803/899522609] [v2:127.0.0.1:6804/899522609,v1:127.0.0.1:6805/899522609] exists 3fba3efd-872b-4973-a110-aa43a417f30f 2026-02-22T22:32:52.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:32:52.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:52.312 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:52.313 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:52.313 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:52.313 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:52.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:52.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:32:52.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:32:52.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:32:52.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:32:52.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:32:52.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:32:52.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:32:52.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:32:52.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-22T22:32:52.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:32:52.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T22:32:52.315 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:32:52.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-22T22:32:52.317 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-22T22:32:52.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T22:32:52.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:32:52.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:32:52.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:32:52.326 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:52.323+0000 7fc5d3cfe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:52.328 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:52.325+0000 7fc5d3cfe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:52.328 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:52.326+0000 7fc5d3cfe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:52.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T22:32:52.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:32:52.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:32:52.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:32:52.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:32:52.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:52.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:32:52.503 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:52.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:52.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:52.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:53.661 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:53.659+0000 7fc5d3cfe780 -1 Falling back to public interface 2026-02-22T22:32:53.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:53.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:53.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:32:53.694 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:32:53.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:53.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:53.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:54.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:54.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:54.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:32:54.881 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:32:54.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:54.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:55.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:56.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:56.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:56.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:32:56.071 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:32:56.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:56.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:56.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:32:56.356 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:56.354+0000 7fc5d3cfe780 -1 osd.0 5 log_to_monitors true 2026-02-22T22:32:57.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:32:57.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:32:57.265 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:32:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:32:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:32:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:32:57.453 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/716170465,v1:127.0.0.1:6803/716170465] [v2:127.0.0.1:6804/716170465,v1:127.0.0.1:6805/716170465] exists,up 3fba3efd-872b-4973-a110-aa43a417f30f 2026-02-22T22:32:57.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:32:57.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:32:57.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:32:57.455 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-22T22:32:57.455 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:32:57.455 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:32:57.456 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:57.456 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.456 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.456 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:32:57.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-22T22:32:57.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:32:57.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2026-02-22T22:32:57.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-22T22:32:57.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:57.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:57.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:57.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:32:57.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:32:57.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:32:57.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:32:57.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:32:57.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:32:57.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:57.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:32:57.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:32:57.616 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:57.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:32:57.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:32:57.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:32:57.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:32:57.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:32:57.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:32:57.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:32:57.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:32:57.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:32:57.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:32:57.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:32:57.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:32:57.626 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:32:57.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:32:57.626 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:32:57.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:32:57.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:32:57.626 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:32:57.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:57.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:32:57.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:32:57.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:32:57.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:32:57.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:32:57.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:32:57.630 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:32:57.630 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.630 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:32:57.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:32:57.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:32:57.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:32:57.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:32:57.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:32:57.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:32:57.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:32:57.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:32:57.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:32:57.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2026-02-22T22:32:57.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:32:57.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:32:57.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:32:57.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:32:57.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:32:57.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:32:57.654 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:32:57.654 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:57.654 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:57.654 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:57.654 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.654 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.655 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:57.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:32:57.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:32:57.682 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:32:57.682 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:32:57.682 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.683 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.684 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:32:57.684 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:32:57.684 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:32:57.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:32:57.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:32:57.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:32:57.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:32:57.770 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2026-02-22T22:32:57.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:32:57.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:32:57.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:32:57.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:32:57.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:32:57.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:32:57.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:32:57.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:57.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:57.869 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:57.869 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.869 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:57.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:32:57.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:32:57.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-22T22:32:57.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:32:57.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:32:57.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3f7ab76-7be4-4960-9686-4606f1692ae4 2026-02-22T22:32:57.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f3f7ab76-7be4-4960-9686-4606f1692ae4' 2026-02-22T22:32:57.894 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 f3f7ab76-7be4-4960-9686-4606f1692ae4 2026-02-22T22:32:57.894 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:32:57.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZhJtpag6mNRAAS2Ly9NbsSCxtC8kplN10iw== 2026-02-22T22:32:57.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZhJtpag6mNRAAS2Ly9NbsSCxtC8kplN10iw=="}' 2026-02-22T22:32:57.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3f7ab76-7be4-4960-9686-4606f1692ae4 -i td/ceph-helpers/0/new.json 2026-02-22T22:32:58.004 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:32:58.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:32:58.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQAZhJtpag6mNRAAS2Ly9NbsSCxtC8kplN10iw== --osd-uuid f3f7ab76-7be4-4960-9686-4606f1692ae4 2026-02-22T22:32:58.030 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:58.029+0000 7f0c9efab780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:58.032 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:58.030+0000 7f0c9efab780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:58.032 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:58.031+0000 7f0c9efab780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:32:58.033 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:58.031+0000 7f0c9efab780 -1 bdev(0x55b4799a5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:32:58.033 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:32:58.031+0000 7f0c9efab780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:33:00.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:33:00.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:33:00.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:33:00.681 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:33:00.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:33:00.936 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:33:00.936 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:33:00.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-22T22:33:00.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:33:00.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:33:00.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:33:00.953 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:00.952+0000 7f1496cfe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:00.955 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:00.953+0000 7f1496cfe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:00.956 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:00.954+0000 7f1496cfe780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:01.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:33:01.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:33:01.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:33:01.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:33:01.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:33:01.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:01.125 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:01.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:33:01.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:01.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:01.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:02.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:02.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:02.313 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:33:02.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:33:02.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:02.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:02.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:02.546 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:02.544+0000 7f1496cfe780 -1 Falling back to public interface 2026-02-22T22:33:03.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:03.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:03.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:33:03.504 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:33:03.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:03.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:03.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:04.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:04.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:04.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:33:04.695 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:33:04.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:04.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:04.885 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:05.493 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:05.492+0000 7f1496cfe780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:33:05.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:05.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:05.886 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:33:05.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:33:05.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:05.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:06.075 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1966710416,v1:127.0.0.1:6803/1966710416] [v2:127.0.0.1:6804/1966710416,v1:127.0.0.1:6805/1966710416] exists,up f3f7ab76-7be4-4960-9686-4606f1692ae4 2026-02-22T22:33:06.076 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-22T22:33:06.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-22T22:33:06.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-22T22:33:06.125 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-22T22:33:06.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-22T22:33:06.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-22T22:33:06.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2026-02-22T22:33:06.173 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:06.173 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:06.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:06.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:06.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:07.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:07.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2026-02-22T22:33:07.672 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 is already down. 2026-02-22T22:33:07.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2026-02-22T22:33:07.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2026-02-22T22:33:07.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:33:07.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:33:07.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:33:07.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:07.682 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:07.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:33:07.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:07.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2026-02-22T22:33:07.869 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1966710416,v1:127.0.0.1:6803/1966710416] [v2:127.0.0.1:6804/1966710416,v1:127.0.0.1:6805/1966710416] exists f3f7ab76-7be4-4960-9686-4606f1692ae4 2026-02-22T22:33:07.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:33:07.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:33:07.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:33:07.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:07.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-22T22:33:07.871 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-22T22:33:07.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T22:33:07.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:33:07.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:33:07.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:33:07.889 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:07.888+0000 7f9f55a93780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:07.890 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:07.889+0000 7f9f55a93780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:07.891 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:07.890+0000 7f9f55a93780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:08.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:08.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:08.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:09.227 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:09.225+0000 7f9f55a93780 -1 Falling back to public interface 2026-02-22T22:33:09.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:09.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:09.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:33:09.252 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:33:09.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:09.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:09.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:10.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:10.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:10.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:33:10.440 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:33:10.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:10.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:10.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:11.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:11.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:11.633 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:33:11.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:33:11.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:11.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:11.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:11.914 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:11.912+0000 7f9f55a93780 -1 osd.0 5 log_to_monitors true 2026-02-22T22:33:12.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:12.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:12.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:33:12.823 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:33:12.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:12.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:13.010 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/2377459468,v1:127.0.0.1:6803/2377459468] [v2:127.0.0.1:6804/2377459468,v1:127.0.0.1:6805/2377459468] exists,up f3f7ab76-7be4-4960-9686-4606f1692ae4 2026-02-22T22:33:13.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:33:13.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:33:13.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:33:13.012 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-22T22:33:13.012 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:33:13.012 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:33:13.012 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:13.013 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.013 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.013 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:33:13.013 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-22T22:33:13.013 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-22T22:33:13.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-22T22:33:13.060 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-22T22:33:13.060 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:33:13.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:33:13.061 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:13.061 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.061 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.061 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:33:13.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-22T22:33:13.061 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-22T22:33:13.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-22T22:33:13.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-22T22:33:13.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-22T22:33:13.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2026-02-22T22:33:13.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:13.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:13.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:13.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:13.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:13.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:13.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:13.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:13.216 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:13.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:13.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:13.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:13.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:13.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:13.219 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:13.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:13.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:13.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:13.221 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:13.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:13.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:13.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:13.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:13.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:13.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:13.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:13.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:13.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_destroy_osd td/ceph-helpers 2026-02-22T22:33:13.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2026-02-22T22:33:13.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2026-02-22T22:33:13.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:13.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:13.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:13.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:13.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:13.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:13.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:13.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:13.231 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:13.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:13.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:13.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:13.233 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:13.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:13.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:13.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:13.235 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:13.235 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.235 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:13.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:13.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:13.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:13.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:13.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:13.238 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:13.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:33:13.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:33:13.258 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:33:13.258 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:13.259 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:13.259 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:13.259 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.259 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.259 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:13.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:33:13.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:33:13.286 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:33:13.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:13.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:13.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:33:13.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:33:13.287 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:13.287 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:13.287 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:13.289 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:13.289 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.289 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.289 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:13.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:13.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:33:13.333 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:33:13.333 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:13.333 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:13.333 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.334 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:13.335 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:13.335 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:33:13.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2026-02-22T22:33:13.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:33:13.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:33:13.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:33:13.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:33:13.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:33:13.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:33:13.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:33:13.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:13.485 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:13.485 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:13.485 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.485 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.485 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:13.485 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:13.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:13.503 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:33:13.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:33:13.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:33:13.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=191de145-46ac-427e-bd4d-24c8e7e2b2b7 2026-02-22T22:33:13.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 191de145-46ac-427e-bd4d-24c8e7e2b2b7' 2026-02-22T22:33:13.505 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 191de145-46ac-427e-bd4d-24c8e7e2b2b7 2026-02-22T22:33:13.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:33:13.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAphJtpzkiyHhAA3RjgnX6ZzIzukbzBfOJXsA== 2026-02-22T22:33:13.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAphJtpzkiyHhAA3RjgnX6ZzIzukbzBfOJXsA=="}' 2026-02-22T22:33:13.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 191de145-46ac-427e-bd4d-24c8e7e2b2b7 -i td/ceph-helpers/0/new.json 2026-02-22T22:33:13.619 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:13.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:33:13.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAphJtpzkiyHhAA3RjgnX6ZzIzukbzBfOJXsA== --osd-uuid 191de145-46ac-427e-bd4d-24c8e7e2b2b7 2026-02-22T22:33:13.644 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:13.642+0000 7fd12400c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:13.646 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:13.644+0000 7fd12400c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:13.647 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:13.645+0000 7fd12400c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:13.647 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:13.645+0000 7fd12400c780 -1 bdev(0x5636bc87fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:33:13.648 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:13.645+0000 7fd12400c780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:33:16.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:33:16.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:33:16.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:33:16.301 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:33:16.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:33:16.558 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:33:16.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:33:16.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:33:16.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:33:16.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:33:16.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:33:16.574 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:16.572+0000 7fc785b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:16.575 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:16.574+0000 7fc785b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:16.576 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:16.574+0000 7fc785b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:16.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:33:16.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:33:16.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:33:16.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:33:16.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:33:16.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:16.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:33:16.748 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:16.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:16.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:16.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:17.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:17.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:17.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:33:17.937 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:33:17.938 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:17.938 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:18.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:18.161 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:18.159+0000 7fc785b19780 -1 Falling back to public interface 2026-02-22T22:33:19.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:19.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:19.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:33:19.127 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:33:19.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:19.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:19.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:20.320 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:20.320 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:20.320 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:33:20.321 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:33:20.321 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:20.321 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:20.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:20.854 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:20.851+0000 7fc785b19780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:33:21.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:21.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:21.513 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:33:21.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:33:21.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:21.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:21.701 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/311458643,v1:127.0.0.1:6803/311458643] [v2:127.0.0.1:6804/311458643,v1:127.0.0.1:6805/311458643] exists,up 191de145-46ac-427e-bd4d-24c8e7e2b2b7 2026-02-22T22:33:21.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:33:21.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:33:21.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:33:21.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2026-02-22T22:33:21.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2026-02-22T22:33:21.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2026-02-22T22:33:21.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2026-02-22T22:33:21.920 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 is already out. 2026-02-22T22:33:21.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2026-02-22T22:33:21.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:21.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:21.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:21.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:21.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:23.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:23.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2026-02-22T22:33:23.426 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 is already down. 2026-02-22T22:33:23.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2026-02-22T22:33:23.619 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 does not exist 2026-02-22T22:33:23.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2026-02-22T22:33:23.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2026-02-22T22:33:23.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:23.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2026-02-22T22:33:23.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:23.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:23.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:23.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:23.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:23.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:23.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:23.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:23.630 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:23.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:23.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:23.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:23.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:23.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:23.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:23.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:23.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:23.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:23.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2026-02-22T22:33:23.636 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:23.636 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:23.636 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:23.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:23.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:23.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:23.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2026-02-22T22:33:23.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2026-02-22T22:33:23.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2026-02-22T22:33:23.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:23.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:23.938 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:23.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:23.939 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:23.939 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:23.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:23.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:23.941 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:23.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:23.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:23.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:23.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:23.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:23.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:23.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:23.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:23.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:23.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:23.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:23.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:23.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:23.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_display_logs td/ceph-helpers 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: test_display_logs: local dir=td/ceph-helpers 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: test_display_logs: setup td/ceph-helpers 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:23.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:23.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:23.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:23.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:23.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:23.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:23.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:23.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:23.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:23.953 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:23.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:23.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:23.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:23.954 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:23.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:23.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:23.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:23.957 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:23.957 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:23.957 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:23.957 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:23.958 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:23.958 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:23.958 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:23.959 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:23.959 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:23.959 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:23.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: test_display_logs: run_mon td/ceph-helpers a 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:33:23.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:33:23.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:33:23.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:33:23.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:33:23.981 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:33:23.981 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:23.981 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:23.981 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:23.981 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:23.981 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:23.981 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:23.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:33:24.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:33:24.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:33:24.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:24.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:24.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:33:24.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:33:24.009 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:24.010 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:24.010 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:24.010 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:24.010 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.010 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.010 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:24.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:24.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:24.057 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:24.058 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:24.058 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.058 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.058 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:24.058 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:24.058 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:33:24.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: test_display_logs: kill_daemons td/ceph-helpers 2026-02-22T22:33:24.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:24.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:24.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:24.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:24.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:24.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:24.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: test_display_logs: display_logs td/ceph-helpers 2026-02-22T22:33:24.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: display_logs: local dir=td/ceph-helpers 2026-02-22T22:33:24.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2026-02-22T22:33:24.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: display_logs: read file 2026-02-22T22:33:24.209 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2026-02-22T22:33:24.209 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: display_logs: cat td/ceph-helpers/mon.a.log 2026-02-22T22:33:24.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: display_logs: read file 2026-02-22T22:33:24.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2026-02-22T22:33:24.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: test_display_logs: teardown td/ceph-helpers 2026-02-22T22:33:24.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:24.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:24.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:24.212 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:24.213 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:24.213 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:24.213 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:24.213 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:24.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:24.214 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:24.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:24.215 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:24.216 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:24.216 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:24.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:24.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:24.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:24.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:24.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:24.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:24.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:24.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:24.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: test_erasure_code_plugin_exists: setup td/ceph-helpers 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:24.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:24.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:24.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:24.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:24.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:24.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:24.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:24.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:24.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:24.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:24.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:24.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:24.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:24.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:24.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:24.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:24.227 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:24.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:24.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:24.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:24.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:24.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:24.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:24.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:24.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:24.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:24.233 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:24.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:24.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:24.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:24.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2026-02-22T22:33:24.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:33:24.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:33:24.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:33:24.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:33:24.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:33:24.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:33:24.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:33:24.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:24.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:24.253 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:24.253 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.254 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.254 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:24.254 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:33:24.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:33:24.279 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:33:24.279 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:24.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:24.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:33:24.281 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:33:24.281 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:24.281 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:24.281 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:24.281 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:24.282 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.282 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.282 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:24.282 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:24.282 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:33:24.326 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.327 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:24.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:24.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:33:24.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2026-02-22T22:33:24.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:33:24.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:33:24.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:33:24.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:33:24.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:33:24.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:33:24.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:33:24.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:24.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:24.472 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:24.472 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.472 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:24.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:24.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2026-02-22T22:33:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local plugin=jerasure 2026-02-22T22:33:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: local status 2026-02-22T22:33:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: local grepstr 2026-02-22T22:33:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: local s 2026-02-22T22:33:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: erasure_code_plugin_exists: case `uname` in 2026-02-22T22:33:24.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: erasure_code_plugin_exists: uname 2026-02-22T22:33:24.491 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2026-02-22T22:33:24.491 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2026-02-22T22:33:24.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: erasure_code_plugin_exists: s= 2026-02-22T22:33:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: erasure_code_plugin_exists: local status=0 2026-02-22T22:33:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-22T22:33:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-22T22:33:24.772 INFO:tasks.workunit.client.0.trial123.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-22T22:33:24.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: erasure_code_plugin_exists: return 0 2026-02-22T22:33:24.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2026-02-22T22:33:24.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local plugin=FAKE 2026-02-22T22:33:24.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: local status 2026-02-22T22:33:24.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: local grepstr 2026-02-22T22:33:24.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: local s 2026-02-22T22:33:24.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: erasure_code_plugin_exists: case `uname` in 2026-02-22T22:33:24.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: erasure_code_plugin_exists: uname 2026-02-22T22:33:24.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2026-02-22T22:33:24.781 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2026-02-22T22:33:24.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2026-02-22T22:33:24.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: erasure_code_plugin_exists: local status=5 2026-02-22T22:33:24.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2026-02-22T22:33:24.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2026-02-22T22:33:24.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2026-02-22T22:33:24.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: erasure_code_plugin_exists: return 5 2026-02-22T22:33:24.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2026-02-22T22:33:24.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:24.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:24.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:24.971 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:24.971 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:24.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:24.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:24.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:24.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:24.974 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:24.975 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.975 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:24.975 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:24.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.976 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:24.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:24.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:24.979 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:24.979 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.979 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.979 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:24.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:24.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:24.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:24.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_expect_failure td/ceph-helpers 2026-02-22T22:33:24.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_expect_failure: local dir=td/ceph-helpers 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: test_expect_failure: setup td/ceph-helpers 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:24.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:24.983 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:24.983 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:24.983 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:24.984 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:24.984 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:24.984 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:24.984 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:24.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:24.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:24.986 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:24.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:24.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:24.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:24.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:24.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:24.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:24.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:24.990 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:24.990 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:24.991 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:24.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:24.991 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:24.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: local dir=td/ceph-helpers 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: expect_failure: shift 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: expect_failure: local expected=FAIL 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: shift 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: local success 2026-02-22T22:33:24.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: expect_failure: bash -c 'echo FAIL ; exit 1' 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: expect_failure: success=false 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: false 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: expect_failure: return 0 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: local dir=td/ceph-helpers 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: expect_failure: shift 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: expect_failure: local expected=FAIL 2026-02-22T22:33:24.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: shift 2026-02-22T22:33:24.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: local success 2026-02-22T22:33:24.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: expect_failure: bash -c 'echo FAIL ; exit 0' 2026-02-22T22:33:24.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: success=true 2026-02-22T22:33:24.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: true 2026-02-22T22:33:24.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2038: expect_failure: cat td/ceph-helpers/out 2026-02-22T22:33:24.997 INFO:tasks.workunit.client.0.trial123.stderr:FAIL 2026-02-22T22:33:24.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: expect_failure: return 1 2026-02-22T22:33:24.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-22T22:33:24.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: local dir=td/ceph-helpers 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: expect_failure: shift 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: expect_failure: local expected=FAIL 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: shift 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: local success 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: expect_failure: success=false 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: false 2026-02-22T22:33:24.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-22T22:33:25.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2038: expect_failure: cat td/ceph-helpers/out 2026-02-22T22:33:25.000 INFO:tasks.workunit.client.0.trial123.stderr:UNEXPECTED 2026-02-22T22:33:25.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: expect_failure: return 1 2026-02-22T22:33:25.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2056: test_expect_failure: teardown td/ceph-helpers 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:25.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:25.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:25.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:25.003 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:25.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:25.004 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:25.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:25.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:25.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:25.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:25.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:25.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:25.007 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:25.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:25.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:25.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:25.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:25.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.010 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_flush_pg_stats td/ceph-helpers 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_flush_pg_stats: local dir=td/ceph-helpers 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: setup td/ceph-helpers 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:25.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:25.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:25.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:25.013 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:25.013 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:25.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:25.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:25.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:25.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:25.014 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:25.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:25.014 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:25.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:25.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:25.015 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:25.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:25.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:25.016 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:25.016 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:25.017 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:25.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:25.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:25.019 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:25.019 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.019 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:25.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:25.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:25.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:25.021 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:25.021 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.021 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:25.022 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:33:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:33:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:33:25.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:33:25.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:33:25.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:25.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:33:25.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:25.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:25.043 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:25.043 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.043 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:25.043 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:25.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:33:25.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:33:25.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:25.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:25.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:33:25.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:33:25.072 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:25.072 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:25.072 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:25.072 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:25.072 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.073 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.073 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:25.073 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:25.073 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:25.120 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:25.121 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.121 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.121 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:25.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:25.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:33:25.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2292: test_flush_pg_stats: run_mgr td/ceph-helpers x 2026-02-22T22:33:25.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:33:25.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:33:25.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:33:25.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:33:25.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:33:25.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:33:25.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:33:25.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:25.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:25.266 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:25.266 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.267 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.267 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:25.267 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:25.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:25.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2293: test_flush_pg_stats: run_osd td/ceph-helpers 0 2026-02-22T22:33:25.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:33:25.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:33:25.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:33:25.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:33:25.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:33:25.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:33:25.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:33:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e4c7b9b-5b92-4147-a2a7-08afe35f1755 2026-02-22T22:33:25.291 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 9e4c7b9b-5b92-4147-a2a7-08afe35f1755 2026-02-22T22:33:25.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9e4c7b9b-5b92-4147-a2a7-08afe35f1755' 2026-02-22T22:33:25.291 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:33:25.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1hJtpKRS5ERAAO4wnh2wLydyOm8njusjyfg== 2026-02-22T22:33:25.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1hJtpKRS5ERAAO4wnh2wLydyOm8njusjyfg=="}' 2026-02-22T22:33:25.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e4c7b9b-5b92-4147-a2a7-08afe35f1755 -i td/ceph-helpers/0/new.json 2026-02-22T22:33:25.401 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:25.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:33:25.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1hJtpKRS5ERAAO4wnh2wLydyOm8njusjyfg== --osd-uuid 9e4c7b9b-5b92-4147-a2a7-08afe35f1755 2026-02-22T22:33:25.425 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:25.422+0000 7f3e7f42c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:25.427 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:25.424+0000 7f3e7f42c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:25.428 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:25.425+0000 7f3e7f42c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:25.428 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:25.425+0000 7f3e7f42c780 -1 bdev(0x55c94c969c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:33:25.428 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:25.425+0000 7f3e7f42c780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:33:28.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:33:28.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:33:28.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:33:28.330 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:33:28.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:33:28.588 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:33:28.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:33:28.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:33:28.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:33:28.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:33:28.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:33:28.604 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:28.601+0000 7faf5eb2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:28.605 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:28.603+0000 7faf5eb2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:28.606 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:28.604+0000 7faf5eb2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:28.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:33:28.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:33:28.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:33:28.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:33:28.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:33:28.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:28.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:33:28.776 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:28.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:28.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:28.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:29.944 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:29.942+0000 7faf5eb2e780 -1 Falling back to public interface 2026-02-22T22:33:29.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:29.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:29.962 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:33:29.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:33:29.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:29.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:30.151 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:31.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:31.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:31.153 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:33:31.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:33:31.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:31.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:31.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:32.341 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:32.342 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:32.342 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:33:32.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:33:32.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:32.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:32.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:32.656 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:32.654+0000 7faf5eb2e780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:33:33.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:33.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:33.533 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:33:33.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:33:33.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:33.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:33.721 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3006631772,v1:127.0.0.1:6803/3006631772] [v2:127.0.0.1:6804/3006631772,v1:127.0.0.1:6805/3006631772] exists,up 9e4c7b9b-5b92-4147-a2a7-08afe35f1755 2026-02-22T22:33:33.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:33:33.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:33:33.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:33:33.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: test_flush_pg_stats: create_rbd_pool 2026-02-22T22:33:33.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:33:33.902 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:33:33.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:33:33.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:33:34.146 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:33:34.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:33:35.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:33:35.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: test_flush_pg_stats: rados -p rbd put obj /etc/group 2026-02-22T22:33:35.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: test_flush_pg_stats: flush_pg_stats 2026-02-22T22:33:35.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:33:35.455 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:33:35.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:33:35.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:33:35.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:33:35.646 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:33:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836484 2026-02-22T22:33:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836484 2026-02-22T22:33:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T22:33:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:33:35.711 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836484 2026-02-22T22:33:35.711 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:33:35.712 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:33:35.712 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836484 2026-02-22T22:33:35.712 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:33:35.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836484 2026-02-22T22:33:35.713 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836484 2026-02-22T22:33:35.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T22:33:35.714 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:33:35.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:33:35.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:33:36.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:33:36.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:33:37.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:33:37.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:33:38.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:33:38.093 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:33:38.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T22:33:38.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2026-02-22T22:33:38.283 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: test_flush_pg_stats: ceph df detail --format=json 2026-02-22T22:33:38.283 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2026-02-22T22:33:38.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: test_flush_pg_stats: stored=8189 2026-02-22T22:33:38.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: test_flush_pg_stats: ceph df detail --format=json 2026-02-22T22:33:38.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2026-02-22T22:33:38.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: test_flush_pg_stats: stored_raw=8189 2026-02-22T22:33:38.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: test_flush_pg_stats: test 8189 -gt 0 2026-02-22T22:33:38.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2301: test_flush_pg_stats: test 8189 == 8189 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2302: test_flush_pg_stats: teardown td/ceph-helpers 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:38.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:38.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:38.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:38.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:38.893 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:38.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:38.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:38.895 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:38.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:38.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:38.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:38.897 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:38.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:38.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:38.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:38.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:38.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:38.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:38.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:38.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:38.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:38.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:38.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_config td/ceph-helpers 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:38.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:38.908 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:38.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:38.908 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:38.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:38.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:38.910 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:38.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:38.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:38.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:38.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:38.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:38.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:38.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:38.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:38.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:38.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:38.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:38.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:38.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:38.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:38.914 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:38.914 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:38.914 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:38.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:38.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:38.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:33:38.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:38.936 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:38.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:38.964 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:38.965 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:38.965 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:38.965 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:38.965 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:38.965 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:33:39.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:39.010 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:39.011 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:39.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:39.011 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:33:39.055 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:39.056 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:39.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:39.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get osd_pool_default_size 2026-02-22T22:33:39.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2026-02-22T22:33:39.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2026-02-22T22:33:39.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:33:39.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:33:39.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:33:39.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:33:39.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:33:39.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:33:39.200 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:33:39.200 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:39.200 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:39.201 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:39.201 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:39.201 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:39.201 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:39.201 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:39.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:33:39.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:39.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:33:39.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17a3a40a-d95a-4c35-80a9-26744602e4b9 2026-02-22T22:33:39.224 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 17a3a40a-d95a-4c35-80a9-26744602e4b9 2026-02-22T22:33:39.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 17a3a40a-d95a-4c35-80a9-26744602e4b9' 2026-02-22T22:33:39.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:33:39.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDhJtpCZncDRAA2U4cZF1O3jmRawb2vSqYPw== 2026-02-22T22:33:39.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDhJtpCZncDRAA2U4cZF1O3jmRawb2vSqYPw=="}' 2026-02-22T22:33:39.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17a3a40a-d95a-4c35-80a9-26744602e4b9 -i td/ceph-helpers/0/new.json 2026-02-22T22:33:39.337 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:39.344 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:33:39.345 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQBDhJtpCZncDRAA2U4cZF1O3jmRawb2vSqYPw== --osd-uuid 17a3a40a-d95a-4c35-80a9-26744602e4b9 2026-02-22T22:33:39.363 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:39.361+0000 7ff29d410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:39.364 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:39.362+0000 7ff29d410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:39.365 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:39.363+0000 7ff29d410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:39.365 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:39.364+0000 7ff29d410780 -1 bdev(0x55d69c6d5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:33:39.366 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:39.364+0000 7ff29d410780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:33:41.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:33:41.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:33:41.521 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:33:41.521 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:33:41.521 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:33:41.772 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:33:41.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:33:41.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2026-02-22T22:33:41.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:33:41.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:33:41.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:33:41.789 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:41.787+0000 7fdb6e53f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:41.790 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:41.788+0000 7fdb6e53f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:41.791 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:41.789+0000 7fdb6e53f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:41.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:33:41.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:33:41.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:33:41.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:33:41.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:33:41.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:41.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:33:41.962 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:41.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:41.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:42.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:43.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:43.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:43.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:33:43.154 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:33:43.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:43.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:43.344 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:43.374 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:43.372+0000 7fdb6e53f780 -1 Falling back to public interface 2026-02-22T22:33:44.345 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:44.346 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:44.346 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:33:44.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:33:44.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:44.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:44.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:45.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:45.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:45.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:33:45.541 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:33:45.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:45.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:45.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:46.324 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:46.322+0000 7fdb6e53f780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:33:46.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:46.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:33:46.733 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:33:46.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:46.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:46.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:47.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:47.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:47.925 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:33:47.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:33:47.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:47.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:48.115 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1965685884,v1:127.0.0.1:6803/1965685884] [v2:127.0.0.1:6804/1965685884,v1:127.0.0.1:6805/1965685884] exists,up 17a3a40a-d95a-4c35-80a9-26744602e4b9 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2026-02-22T22:33:48.118 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:48.119 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_max_scrubs 2026-02-22T22:33:48.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2026-02-22T22:33:48.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2026-02-22T22:33:48.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:48.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:48.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:48.165 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:48.165 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:48.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:48.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:48.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:48.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:48.272 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:48.273 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:48.273 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:48.274 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:48.274 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:48.274 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:48.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:48.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:48.276 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:48.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:48.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:48.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:48.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.281 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:48.282 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:48.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:48.284 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:48.284 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:48.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:48.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:48.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:48.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:48.285 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:48.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:48.285 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:48.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:48.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:48.286 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:48.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:48.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:48.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:48.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:48.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:48.290 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:48.290 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.290 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:48.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:48.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:48.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:48.293 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:48.293 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.293 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:48.296 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:48.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:48.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:48.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:48.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2026-02-22T22:33:48.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:33:48.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:33:48.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:33:48.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:33:48.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:33:48.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:33:48.314 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:33:48.315 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:48.315 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:48.315 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:48.315 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.315 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.315 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:48.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:33:48.341 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:48.343 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:48.344 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.344 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.344 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:48.344 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:48.344 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:33:48.389 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:48.390 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.391 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.391 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:48.391 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:48.391 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:33:48.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2026-02-22T22:33:48.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:33:48.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:33:48.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:33:48.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:33:48.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:33:48.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:33:48.535 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:33:48.535 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:48.535 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:48.535 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:48.535 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.536 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:48.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:48.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:48.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2026-02-22T22:33:48.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:33:48.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:33:48.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:33:48.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:33:48.554 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:33:48.554 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:33:48.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:33:48.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:33:48.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2026-02-22T22:33:48.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:48.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:48.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:48.658 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:48.658 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:48.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:48.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:48.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:48.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:48.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:48.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:48.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:48.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:48.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:48.766 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:48.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:48.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:48.767 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:48.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:48.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:48.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:48.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:48.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:48.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:48.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:33:48.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2026-02-22T22:33:48.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2026-02-22T22:33:48.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:33:48.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:33:48.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:33:48.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:33:48.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:33:48.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:33:48.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:33:48.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:33:48.776 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:33:48.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:33:48.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:33:48.778 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:33:48.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:33:48.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:33:48.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:33:48.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:33:48.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.781 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:33:48.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:33:48.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:33:48.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:33:48.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:33:48.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:33:48.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:33:48.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:33:48.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:33:48.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:33:48.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:48.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:33:48.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:33:48.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:33:48.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:33:48.788 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:33:48.788 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:48.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:33:48.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:48.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:48.806 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:48.806 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.806 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.807 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:48.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:33:48.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:33:48.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:33:48.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:48.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:48.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:33:48.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:33:48.833 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:48.833 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:48.833 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:48.834 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:48.834 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.834 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.834 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:48.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:48.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:33:48.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:33:48.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:33:48.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:33:48.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:33:48.878 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:33:48.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:33:48.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2026-02-22T22:33:48.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:33:48.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:33:48.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:33:48.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:33:48.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:33:48.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:33:49.024 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:33:49.024 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:49.024 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:49.025 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:49.025 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:49.025 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:49.025 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:49.025 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:49.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:33:49.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2026-02-22T22:33:49.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:33:49.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:33:49.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:33:49.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9c88afb-58c1-4996-a08e-3eaafbb1d831 2026-02-22T22:33:49.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d9c88afb-58c1-4996-a08e-3eaafbb1d831' 2026-02-22T22:33:49.050 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 d9c88afb-58c1-4996-a08e-3eaafbb1d831 2026-02-22T22:33:49.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:33:49.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNhJtpK3NvAxAAG3rtp4dkaq7H8k8uMxvfPQ== 2026-02-22T22:33:49.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNhJtpK3NvAxAAG3rtp4dkaq7H8k8uMxvfPQ=="}' 2026-02-22T22:33:49.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9c88afb-58c1-4996-a08e-3eaafbb1d831 -i td/ceph-helpers/0/new.json 2026-02-22T22:33:49.161 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:49.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:33:49.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNhJtpK3NvAxAAG3rtp4dkaq7H8k8uMxvfPQ== --osd-uuid d9c88afb-58c1-4996-a08e-3eaafbb1d831 2026-02-22T22:33:49.187 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:49.186+0000 7f5b03184780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:49.189 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:49.187+0000 7f5b03184780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:49.190 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:49.188+0000 7f5b03184780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:49.190 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:49.188+0000 7f5b03184780 -1 bdev(0x560361089c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:33:49.190 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:49.189+0000 7f5b03184780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:33:51.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:33:51.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:33:51.836 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:33:51.836 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:33:51.836 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:33:52.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:33:52.092 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:33:52.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:33:52.093 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:33:52.093 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:33:52.093 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:33:52.108 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:52.106+0000 7ff5233d0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:52.109 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:52.107+0000 7ff5233d0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:52.110 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:52.108+0000 7ff5233d0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:33:52.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:33:52.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:33:52.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:33:52.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:33:52.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:33:52.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:52.278 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:33:52.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:33:52.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:52.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:52.466 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:53.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:53.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:53.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:33:53.468 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:33:53.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:53.468 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:53.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:53.697 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:53.695+0000 7ff5233d0780 -1 Falling back to public interface 2026-02-22T22:33:54.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:54.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:54.659 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:33:54.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:33:54.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:54.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:54.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:55.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:55.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:55.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:33:55.847 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:33:55.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:55.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:56.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:56.630 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:33:56.627+0000 7ff5233d0780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:33:57.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:57.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:57.037 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:33:57.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:33:57.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:57.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:57.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:33:58.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:33:58.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:33:58.231 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:33:58.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:33:58.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:33:58.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:33:58.417 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2747696375,v1:127.0.0.1:6803/2747696375] [v2:127.0.0.1:6804/2747696375,v1:127.0.0.1:6805/2747696375] exists,up d9c88afb-58c1-4996-a08e-3eaafbb1d831 2026-02-22T22:33:58.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:33:58.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:33:58.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:33:58.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2026-02-22T22:33:58.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:33:58.603 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:33:58.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:33:58.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:33:58.848 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:33:58.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:33:59.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:34:00.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2026-02-22T22:34:00.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:34:00.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:34:00.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:34:00.138 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:34:00.138 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:34:00.138 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:34:00.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:34:00.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:34:00.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:34:00.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:34:00.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:34:00.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:34:00.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:34:00.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:34:00.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:34:00.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:34:00.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:34:00.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:34:00.409 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:34:00.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:34:00.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:34:00.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:34:00.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:34:00.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:34:00.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:34:00.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:34:00.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:34:00.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:34:00.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:34:00.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:34:00.476 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:34:00.476 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:00.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:34:00.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:34:01.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:34:01.663 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:01.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:34:01.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:34:01.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:01.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:02.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:34:02.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:34:02.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:34:02.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:34:02.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:34:02.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:34:02.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:34:02.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:34:02.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:34:02.293 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:34:02.294 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:02.294 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:02.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:34:02.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:34:02.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:34:02.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2026-02-22T22:34:02.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:34:02.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:34:02.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:34:02.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:34:02.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2026-02-22T22:33:58.797740+0000 2026-02-22T22:34:02.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2026-02-22T22:33:58.797740+0000 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:02.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:02.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:02.846 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:02.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:02.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:02.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:02.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:02.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:02.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:02.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:02.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:02.850 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:02.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:02.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:02.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:02.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:02.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:02.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:02.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_not_primary td/ceph-helpers 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:34:02.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:02.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:02.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:02.860 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:02.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:02.860 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:02.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:02.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:02.861 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:02.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:02.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:02.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:02.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:02.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:02.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:02.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:02.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:02.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:02.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:02.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:02.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:02.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:02.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:34:02.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:34:02.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:02.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:02.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:34:02.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:02.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T22:34:02.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:34:02.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:34:02.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:02.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:02.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:34:02.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:02.917 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:02.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:34:02.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:34:02.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:02.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:02.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:34:02.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:34:02.963 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:02.963 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:02.963 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:02.963 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:02.963 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:02.963 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:02.963 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:02.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:02.964 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:34:03.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2026-02-22T22:34:03.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:34:03.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:34:03.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:34:03.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:34:03.006 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:34:03.007 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:34:03.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:34:03.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:03.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:03.102 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:03.103 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:03.103 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:03.103 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:03.103 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:03.104 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:34:03.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:34:03.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:34:03.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:34:03.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:34:03.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:34:03.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:34:03.125 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:34:03.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd59c0ea-7998-4035-a02f-2589f01c685b 2026-02-22T22:34:03.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fd59c0ea-7998-4035-a02f-2589f01c685b' 2026-02-22T22:34:03.126 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 fd59c0ea-7998-4035-a02f-2589f01c685b 2026-02-22T22:34:03.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:34:03.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbhJtpqaH0BxAAlfR15kP6V/VdRE9i7q2hdQ== 2026-02-22T22:34:03.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbhJtpqaH0BxAAlfR15kP6V/VdRE9i7q2hdQ=="}' 2026-02-22T22:34:03.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd59c0ea-7998-4035-a02f-2589f01c685b -i td/ceph-helpers/0/new.json 2026-02-22T22:34:03.236 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:03.242 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:34:03.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbhJtpqaH0BxAAlfR15kP6V/VdRE9i7q2hdQ== --osd-uuid fd59c0ea-7998-4035-a02f-2589f01c685b 2026-02-22T22:34:03.260 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:03.257+0000 7f9e772d0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:03.261 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:03.259+0000 7f9e772d0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:03.262 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:03.260+0000 7f9e772d0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:03.263 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:03.260+0000 7f9e772d0780 -1 bdev(0x55d4871bbc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:34:03.263 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:03.260+0000 7f9e772d0780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:34:06.142 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:34:06.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:34:06.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:34:06.144 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:34:06.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:34:06.401 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:34:06.402 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:34:06.402 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:34:06.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:34:06.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:34:06.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:34:06.418 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:06.416+0000 7f89e96ba780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:06.419 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:06.418+0000 7f89e96ba780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:06.420 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:06.419+0000 7f89e96ba780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:06.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:34:06.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:34:06.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:34:06.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:34:06.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:34:06.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:06.589 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:06.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:34:06.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:06.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:06.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:07.763 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:07.761+0000 7f89e96ba780 -1 Falling back to public interface 2026-02-22T22:34:07.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:07.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:07.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:34:07.776 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:34:07.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:07.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:07.966 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:08.968 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:08.968 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:08.968 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:34:08.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:34:08.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:08.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:09.157 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:10.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:10.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:10.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:34:10.159 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:34:10.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:10.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:10.344 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:10.714 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:10.712+0000 7f89e96ba780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:34:11.345 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:11.345 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:11.346 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:34:11.346 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:34:11.346 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:11.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:11.537 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1580660791,v1:127.0.0.1:6803/1580660791] [v2:127.0.0.1:6804/1580660791,v1:127.0.0.1:6805/1580660791] exists,up fd59c0ea-7998-4035-a02f-2589f01c685b 2026-02-22T22:34:11.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:34:11.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:34:11.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:34:11.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2026-02-22T22:34:11.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:34:11.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:34:11.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:34:11.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:34:11.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=581743f4-f846-43c9-a3d4-4d406135fbe9 2026-02-22T22:34:11.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 581743f4-f846-43c9-a3d4-4d406135fbe9' 2026-02-22T22:34:11.546 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 581743f4-f846-43c9-a3d4-4d406135fbe9 2026-02-22T22:34:11.546 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:34:11.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjhJtpmeHYIBAAueyxxFUjvCpVeK8uHyOA1A== 2026-02-22T22:34:11.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjhJtpmeHYIBAAueyxxFUjvCpVeK8uHyOA1A=="}' 2026-02-22T22:34:11.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 581743f4-f846-43c9-a3d4-4d406135fbe9 -i td/ceph-helpers/1/new.json 2026-02-22T22:34:11.745 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:34:11.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:34:11.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjhJtpmeHYIBAAueyxxFUjvCpVeK8uHyOA1A== --osd-uuid 581743f4-f846-43c9-a3d4-4d406135fbe9 2026-02-22T22:34:11.770 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:11.768+0000 7fc84203f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:11.771 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:11.769+0000 7fc84203f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:11.772 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:11.770+0000 7fc84203f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:11.773 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:11.770+0000 7fc84203f780 -1 bdev(0x55a0646d3c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:34:11.773 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:11.771+0000 7fc84203f780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:34:13.885 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:34:13.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:34:13.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:34:13.887 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:34:13.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:34:14.144 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:34:14.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:34:14.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:34:14.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:34:14.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:34:14.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:34:14.161 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:14.159+0000 7fbe77b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:14.162 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:14.160+0000 7fbe77b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:14.163 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:14.161+0000 7fbe77b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:14.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:34:14.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:34:14.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:34:14.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:34:14.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:34:14.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:14.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:34:14.337 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:14.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:14.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:34:14.522 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:15.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:15.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:15.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:34:15.525 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:34:15.526 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:15.526 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:34:15.712 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:15.743 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:15.741+0000 7fbe77b19780 -1 Falling back to public interface 2026-02-22T22:34:16.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:16.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:16.714 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:34:16.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:34:16.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:16.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:34:16.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:17.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:17.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:17.907 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:34:17.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:34:17.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:17.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:34:18.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:18.964 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:18.962+0000 7fbe77b19780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:34:19.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:19.102 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:19.103 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:34:19.103 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:34:19.103 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:19.104 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:34:19.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:20.295 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:20.295 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:20.296 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:34:20.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:34:20.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:20.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:34:20.485 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3725131787,v1:127.0.0.1:6811/3725131787] [v2:127.0.0.1:6812/3725131787,v1:127.0.0.1:6813/3725131787] exists,up 581743f4-f846-43c9-a3d4-4d406135fbe9 2026-02-22T22:34:20.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:34:20.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:34:20.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:34:20.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2026-02-22T22:34:20.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:34:20.670 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:34:20.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:34:20.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:34:20.914 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:34:20.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:34:21.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:34:22.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2026-02-22T22:34:22.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:34:22.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:34:22.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:34:22.205 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:34:22.205 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:34:22.205 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:34:22.205 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:34:22.205 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:34:22.205 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:34:22.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:34:22.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:34:22.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:34:22.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:34:22.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:34:22.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:34:22.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:34:22.476 INFO:tasks.workunit.client.0.trial123.stderr:1' 2026-02-22T22:34:22.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:34:22.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:34:22.477 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:34:22.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836486 2026-02-22T22:34:22.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836486 2026-02-22T22:34:22.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T22:34:22.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:34:22.541 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:34:22.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=38654705667 2026-02-22T22:34:22.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 38654705667 2026-02-22T22:34:22.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486 1-38654705667' 2026-02-22T22:34:22.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:34:22.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836486 2026-02-22T22:34:22.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:34:22.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:34:22.606 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836486 2026-02-22T22:34:22.606 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:34:22.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836486 2026-02-22T22:34:22.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T22:34:22.607 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836486 2026-02-22T22:34:22.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:22.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836486 2026-02-22T22:34:22.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:34:23.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:34:23.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:23.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T22:34:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:34:23.996 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-38654705667 2026-02-22T22:34:23.996 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:34:23.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:34:23.997 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-38654705667 2026-02-22T22:34:23.997 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:34:23.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=38654705667 2026-02-22T22:34:23.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T22:34:23.999 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 38654705667 2026-02-22T22:34:23.999 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:34:24.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T22:34:24.187 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:34:24.188 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:24.188 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:24.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:34:24.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:34:24.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:34:24.444 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:34:24.444 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:34:24.444 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:34:24.444 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:34:24.444 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:34:24.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:34:24.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:34:24.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:24.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2026-02-22T22:34:24.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2026-02-22T22:34:25.078 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2026-02-22T22:34:25.079 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2026-02-22T22:34:25.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2026-02-22T22:34:25.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2026-02-22T22:34:25.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T22:34:25.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2026-02-22T22:34:25.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2026-02-22T22:34:25.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:25.465 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:25.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:25.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:25.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:25.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:25.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:25.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:25.575 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:25.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:25.576 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:25.576 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:25.576 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:25.577 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:25.578 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:25.578 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:25.583 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:25.583 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:25.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_num_active_clean td/ceph-helpers 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:25.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:25.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:25.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:25.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:25.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:25.587 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:25.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:25.588 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:25.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:25.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:25.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:25.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:25.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:25.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:25.591 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:25.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:25.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:25.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:25.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:25.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:25.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:25.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:25.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:34:25.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:34:25.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:34:25.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:34:25.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:34:25.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:25.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:25.644 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:34:25.645 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:34:25.645 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:25.645 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:25.645 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:34:25.645 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:34:25.645 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:25.646 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:25.646 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:25.647 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:25.647 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.647 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.647 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:25.647 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:25.647 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:34:25.690 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.691 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.692 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:25.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:25.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:34:25.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2026-02-22T22:34:25.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:34:25.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:34:25.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:34:25.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:34:25.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:34:25.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:34:25.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:34:25.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:25.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:25.832 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:25.832 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.832 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:25.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:25.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:25.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2026-02-22T22:34:25.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:34:25.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:34:25.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:34:25.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:34:25.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:25.854 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:34:25.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:34:25.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:34:25.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:34:25.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:34:25.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:34:25.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:34:25.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3bc2c7aa-34fb-43e5-b8a9-915e16c3b1f5 2026-02-22T22:34:25.857 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 3bc2c7aa-34fb-43e5-b8a9-915e16c3b1f5 2026-02-22T22:34:25.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3bc2c7aa-34fb-43e5-b8a9-915e16c3b1f5' 2026-02-22T22:34:25.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:34:25.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxhJtpKUZ9MxAAeU8BPF6xtjVEf4iEIvWBFw== 2026-02-22T22:34:25.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxhJtpKUZ9MxAAeU8BPF6xtjVEf4iEIvWBFw=="}' 2026-02-22T22:34:25.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3bc2c7aa-34fb-43e5-b8a9-915e16c3b1f5 -i td/ceph-helpers/0/new.json 2026-02-22T22:34:25.970 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:25.977 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:34:25.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxhJtpKUZ9MxAAeU8BPF6xtjVEf4iEIvWBFw== --osd-uuid 3bc2c7aa-34fb-43e5-b8a9-915e16c3b1f5 2026-02-22T22:34:25.995 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:25.993+0000 7f1b90c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:25.996 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:25.995+0000 7f1b90c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:25.997 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:25.995+0000 7f1b90c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:25.997 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:25.996+0000 7f1b90c2c780 -1 bdev(0x55b123875c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:34:25.998 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:25.996+0000 7f1b90c2c780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:34:28.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:34:28.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:34:28.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:34:28.395 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:34:28.395 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:34:28.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:34:28.653 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:34:28.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:34:28.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:34:28.654 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:34:28.654 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:34:28.669 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:28.667+0000 7f81c4e12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:28.670 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:28.668+0000 7f81c4e12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:28.671 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:28.669+0000 7f81c4e12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:28.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:34:28.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:34:28.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:34:28.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:34:28.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:34:28.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:28.843 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:28.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:34:28.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:28.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:29.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:30.019 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:30.017+0000 7f81c4e12780 -1 Falling back to public interface 2026-02-22T22:34:30.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:30.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:30.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:34:30.030 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:34:30.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:30.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:30.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:31.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:31.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:31.221 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:34:31.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:34:31.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:31.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:32.412 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:32.412 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:32.412 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:34:32.412 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:34:32.413 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:32.413 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:32.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:32.716 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:32.713+0000 7f81c4e12780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:34:33.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:33.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:33.605 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:34:33.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:34:33.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:33.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:33.796 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1929318373,v1:127.0.0.1:6803/1929318373] [v2:127.0.0.1:6804/1929318373,v1:127.0.0.1:6805/1929318373] exists,up 3bc2c7aa-34fb-43e5-b8a9-915e16c3b1f5 2026-02-22T22:34:33.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:34:33.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:34:33.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:34:33.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2026-02-22T22:34:33.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:34:33.979 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:34:33.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:34:33.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:34:34.224 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:34:34.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:34:35.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:34:35.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2026-02-22T22:34:35.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:34:35.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:34:35.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:34:35.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:34:35.513 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:34:35.513 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:34:35.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:34:35.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:34:35.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:34:35.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:34:35.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:34:35.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:34:35.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:34:35.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:34:35.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:34:35.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:34:35.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:34:35.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:34:35.782 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:34:35.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:34:35.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:34:35.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:34:35.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:34:35.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:34:35.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:34:35.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:34:35.848 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:34:35.848 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:34:35.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:34:35.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:34:35.850 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:34:35.850 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:36.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:34:36.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:34:37.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:34:37.040 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:37.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:34:37.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:34:38.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:34:38.233 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:38.422 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:34:38.422 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:34:38.422 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:38.422 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:38.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:34:38.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:34:38.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:34:38.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:34:38.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:34:38.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:34:38.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:34:38.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:34:38.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:34:38.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:34:38.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:38.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:39.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:34:39.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:34:39.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:34:39.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2026-02-22T22:34:39.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:34:39.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:34:39.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:34:39.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:34:39.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:34:39.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2026-02-22T22:34:39.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2026-02-22T22:34:39.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2026-02-22T22:34:39.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:39.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:39.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:39.312 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:39.312 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:39.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:39.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:39.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:39.417 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:39.418 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:39.418 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:39.419 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:39.420 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:39.420 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:39.420 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:39.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:39.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:39.422 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:39.422 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:39.422 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:39.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:39.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:39.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:39.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:39.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:39.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:39.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_num_active_or_peered td/ceph-helpers 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:39.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:39.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:39.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:39.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:39.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:39.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:39.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:39.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:39.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:39.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:39.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:39.434 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:39.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:39.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:39.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:39.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:39.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:39.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:39.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:39.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:34:39.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:34:39.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:34:39.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:34:39.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:34:39.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:39.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:34:39.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:39.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:39.461 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:39.461 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.461 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.461 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:39.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:39.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:34:39.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:34:39.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:39.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:39.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:34:39.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:39.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:34:39.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:34:39.534 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:39.534 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:39.534 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:34:39.534 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:34:39.535 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:39.535 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:39.535 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:39.535 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:39.535 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.535 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.536 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:39.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:39.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:34:39.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2026-02-22T22:34:39.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:34:39.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:34:39.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:34:39.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:34:39.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:34:39.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:34:39.681 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:34:39.681 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:39.681 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:39.681 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:39.681 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.681 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.681 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:39.682 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:39.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:39.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:34:39.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:39.703 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:34:39.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:34:39.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:34:39.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:34:39.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:34:39.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:34:39.705 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:34:39.705 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:34:39.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf8499ce-ea11-4eee-8d03-d5893e5d02d5 2026-02-22T22:34:39.706 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 cf8499ce-ea11-4eee-8d03-d5893e5d02d5 2026-02-22T22:34:39.706 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cf8499ce-ea11-4eee-8d03-d5893e5d02d5' 2026-02-22T22:34:39.707 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:34:39.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/hJtpp0iAKhAA84+91geX0OS2sGGGUQ2c9Q== 2026-02-22T22:34:39.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/hJtpp0iAKhAA84+91geX0OS2sGGGUQ2c9Q=="}' 2026-02-22T22:34:39.715 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf8499ce-ea11-4eee-8d03-d5893e5d02d5 -i td/ceph-helpers/0/new.json 2026-02-22T22:34:39.821 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:39.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:34:39.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/hJtpp0iAKhAA84+91geX0OS2sGGGUQ2c9Q== --osd-uuid cf8499ce-ea11-4eee-8d03-d5893e5d02d5 2026-02-22T22:34:39.847 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:39.844+0000 7f085c5a3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:39.848 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:39.845+0000 7f085c5a3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:39.849 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:39.846+0000 7f085c5a3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:39.850 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:39.846+0000 7f085c5a3780 -1 bdev(0x55eac2dc5c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:34:39.850 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:39.847+0000 7f085c5a3780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:34:42.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:34:42.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:34:42.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:34:42.738 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:34:42.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:34:42.992 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:34:42.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:34:42.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:34:42.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:34:42.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:34:42.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:34:43.008 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:43.007+0000 7fc98e482780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:43.010 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:43.008+0000 7fc98e482780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:43.011 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:43.009+0000 7fc98e482780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:43.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:34:43.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:34:43.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:34:43.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:34:43.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:34:43.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:43.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:34:43.182 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:43.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:43.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:43.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:44.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:44.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:44.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:34:44.369 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:34:44.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:44.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:44.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:44.594 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:44.593+0000 7fc98e482780 -1 Falling back to public interface 2026-02-22T22:34:45.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:45.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:45.559 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:34:45.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:34:45.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:45.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:46.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:46.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:46.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:34:46.751 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:34:46.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:46.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:46.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:47.549 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:47.547+0000 7fc98e482780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:34:47.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:47.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:47.943 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:34:47.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:34:47.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:47.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:48.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:49.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:49.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:49.131 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:34:49.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:34:49.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:49.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:49.318 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2757072313,v1:127.0.0.1:6803/2757072313] [v2:127.0.0.1:6804/2757072313,v1:127.0.0.1:6805/2757072313] exists,up cf8499ce-ea11-4eee-8d03-d5893e5d02d5 2026-02-22T22:34:49.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:34:49.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:34:49.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:34:49.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2026-02-22T22:34:49.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:34:49.503 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:34:49.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:34:49.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:34:49.747 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:34:49.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:34:50.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:34:51.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2026-02-22T22:34:51.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:34:51.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:34:51.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:34:51.036 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:34:51.036 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:34:51.036 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:34:51.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:34:51.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:34:51.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:34:51.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:34:51.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:34:51.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:34:51.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:34:51.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:34:51.119 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:34:51.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:34:51.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:34:51.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:34:51.308 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:34:51.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:34:51.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:34:51.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:34:51.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:34:51.371 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:34:51.371 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:34:51.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:34:51.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:34:51.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:34:51.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:34:51.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:34:51.374 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:34:51.374 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:51.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:34:51.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:34:52.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:34:52.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:34:52.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:34:52.751 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:34:52.751 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:52.751 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:53.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:34:53.009 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:34:53.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:34:53.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:34:53.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:34:53.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:34:53.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:34:53.010 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:34:53.200 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:34:53.200 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:34:53.200 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:34:53.201 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:34:53.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:34:53.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:34:53.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:34:53.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2026-02-22T22:34:53.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2026-02-22T22:34:53.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-22T22:34:53.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-22T22:34:53.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-22T22:34:53.450 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-22T22:34:53.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2026-02-22T22:34:53.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2026-02-22T22:34:53.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2026-02-22T22:34:53.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:53.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:53.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:53.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:53.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:53.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:53.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:53.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:53.742 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:53.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:53.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:53.744 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:53.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:53.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:53.746 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:53.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:53.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:53.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:53.747 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:53.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:53.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:53.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:53.752 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:53.753 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:53.753 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:53.753 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_num_pgs td/ceph-helpers 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:34:53.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:34:53.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:34:53.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:34:53.756 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:34:53.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:34:53.757 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:34:53.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:34:53.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:34:53.758 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:34:53.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:53.759 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:34:53.759 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:34:53.759 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:34:53.760 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:34:53.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:34:53.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:34:53.762 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:34:53.762 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:53.762 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:53.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:34:53.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:34:53.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:34:53.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:34:53.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:34:53.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:53.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:53.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:34:53.765 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:34:53.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:34:53.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:53.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:34:53.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:53.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:53.786 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:53.786 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:53.786 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:53.787 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:53.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:34:53.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:34:53.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:53.814 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:53.815 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:53.815 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:53.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:53.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:34:53.859 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:34:53.859 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:34:53.859 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:34:53.859 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:34:53.859 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:34:53.859 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:34:53.860 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:34:53.860 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:34:53.861 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:34:53.861 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:53.861 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:53.861 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:34:53.861 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:34:53.861 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:34:53.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2026-02-22T22:34:53.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:34:53.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:34:53.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:34:53.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:34:53.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:34:53.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:34:54.003 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:34:54.003 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:54.003 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:54.004 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:54.004 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:54.004 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:54.004 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:54.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:54.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:34:54.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2026-02-22T22:34:54.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:34:54.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:34:54.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:34:54.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:34:54.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:34:54.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:34:54.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3726fabf-c520-4354-9464-62f0533086d1 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3726fabf-c520-4354-9464-62f0533086d1' 2026-02-22T22:34:54.028 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 3726fabf-c520-4354-9464-62f0533086d1 2026-02-22T22:34:54.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:34:54.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOhJtpGM8gAhAAWmGFrPpK7YFa9W26t6lUdQ== 2026-02-22T22:34:54.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOhJtpGM8gAhAAWmGFrPpK7YFa9W26t6lUdQ=="}' 2026-02-22T22:34:54.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3726fabf-c520-4354-9464-62f0533086d1 -i td/ceph-helpers/0/new.json 2026-02-22T22:34:54.140 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:54.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:34:54.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOhJtpGM8gAhAAWmGFrPpK7YFa9W26t6lUdQ== --osd-uuid 3726fabf-c520-4354-9464-62f0533086d1 2026-02-22T22:34:54.165 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:54.164+0000 7ff4e762c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:54.167 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:54.165+0000 7ff4e762c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:54.168 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:54.166+0000 7ff4e762c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:54.168 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:54.166+0000 7ff4e762c780 -1 bdev(0x55631e531c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:34:54.168 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:54.166+0000 7ff4e762c780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:34:56.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:34:56.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:34:56.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:34:56.819 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:34:56.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:34:57.076 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:34:57.076 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:34:57.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:34:57.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:34:57.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:34:57.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:34:57.092 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:57.091+0000 7f08ae210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:57.094 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:57.092+0000 7f08ae210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:57.095 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:57.093+0000 7f08ae210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:34:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:34:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:34:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:34:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:34:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:34:57.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:57.267 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:34:57.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:34:57.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:57.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:57.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:58.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:58.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:58.457 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:34:58.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:34:58.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:58.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:58.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:34:58.676 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:34:58.674+0000 7f08ae210780 -1 Falling back to public interface 2026-02-22T22:34:59.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:34:59.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:34:59.646 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:34:59.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:34:59.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:34:59.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:34:59.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:00.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:00.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:00.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:35:00.838 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:35:00.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:00.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:01.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:01.385 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:01.384+0000 7f08ae210780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:35:02.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:02.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:02.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:35:02.029 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:35:02.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:02.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:02.217 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1884007751,v1:127.0.0.1:6803/1884007751] [v2:127.0.0.1:6804/1884007751,v1:127.0.0.1:6805/1884007751] exists,up 3726fabf-c520-4354-9464-62f0533086d1 2026-02-22T22:35:02.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:35:02.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:35:02.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:35:02.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2026-02-22T22:35:02.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:35:02.400 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:35:02.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:35:02.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:35:02.637 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:35:02.644 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:35:03.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:35:03.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2026-02-22T22:35:03.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:35:03.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:35:03.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:35:03.930 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:35:03.930 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:35:03.930 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:35:03.930 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:35:03.930 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:35:03.930 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:35:04.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:35:04.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:35:04.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:35:04.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:35:04.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:35:04.012 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:35:04.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:35:04.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:35:04.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:35:04.202 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:35:04.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:35:04.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:35:04.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:35:04.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:35:04.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:35:04.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:35:04.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:35:04.267 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:35:04.267 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:35:04.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:35:04.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:35:04.268 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:35:04.268 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:04.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:35:04.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:35:05.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:35:05.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:05.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:35:05.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:35:06.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:35:06.650 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:06.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:35:06.840 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:35:06.840 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:35:06.841 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:35:07.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:35:07.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:35:07.092 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:35:07.092 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:35:07.092 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:35:07.092 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:35:07.092 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:35:07.092 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:35:07.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:35:07.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:35:07.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:35:07.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:35:07.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:35:07.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:35:07.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:35:07.532 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2026-02-22T22:35:07.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:35:07.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:35:07.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2026-02-22T22:35:07.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:35:07.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:35:07.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:35:07.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:35:07.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:35:07.890 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:35:07.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:35:07.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:35:07.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:35:07.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:07.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:35:07.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:35:07.894 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:07.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:35:07.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:35:07.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:35:07.899 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:35:07.899 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:07.899 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:07.899 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_op_scheduler td/ceph-helpers 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: test_get_op_scheduler: local dir=td/ceph-helpers 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: test_get_op_scheduler: setup td/ceph-helpers 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:35:07.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:35:07.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:35:07.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:35:07.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:35:07.903 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:35:07.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:35:07.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:35:07.904 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:35:07.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:07.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:35:07.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:35:07.906 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:07.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:35:07.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:35:07.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:35:07.908 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:35:07.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:07.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:07.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:35:07.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:35:07.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:35:07.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:35:07.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:35:07.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:07.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:07.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:35:07.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:35:07.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: test_get_op_scheduler: run_mon td/ceph-helpers a 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:35:07.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:35:07.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:35:07.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:07.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:07.933 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:07.933 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:07.933 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:07.934 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:07.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:35:07.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:35:07.960 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:35:07.960 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:35:07.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:35:07.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:35:07.961 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:35:07.961 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:35:07.961 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:35:07.961 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:35:07.961 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:07.962 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:07.962 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:07.962 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:35:07.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:07.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:35:08.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:35:08.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:35:08.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:35:08.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:35:08.006 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:08.007 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:35:08.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: test_get_op_scheduler: run_mgr td/ceph-helpers x 2026-02-22T22:35:08.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:35:08.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:35:08.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:35:08.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:35:08.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:35:08.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:35:08.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:35:08.152 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:08.152 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:08.152 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:08.152 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:08.152 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:08.152 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:08.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:35:08.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:35:08.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:08.175 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:35:08.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=970c8b92-195a-4d1e-8372-a088a7c5c01c 2026-02-22T22:35:08.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 970c8b92-195a-4d1e-8372-a088a7c5c01c' 2026-02-22T22:35:08.178 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 970c8b92-195a-4d1e-8372-a088a7c5c01c 2026-02-22T22:35:08.178 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:35:08.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCchJtpcnHuChAAqd1drkhqGS+itl8QcIJURw== 2026-02-22T22:35:08.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCchJtpcnHuChAAqd1drkhqGS+itl8QcIJURw=="}' 2026-02-22T22:35:08.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 970c8b92-195a-4d1e-8372-a088a7c5c01c -i td/ceph-helpers/0/new.json 2026-02-22T22:35:08.286 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:08.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:35:08.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQCchJtpcnHuChAAqd1drkhqGS+itl8QcIJURw== --osd-uuid 970c8b92-195a-4d1e-8372-a088a7c5c01c 2026-02-22T22:35:08.311 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:08.309+0000 7f176b93f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:08.312 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:08.310+0000 7f176b93f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:08.313 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:08.311+0000 7f176b93f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:08.313 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:08.311+0000 7f176b93f780 -1 bdev(0x55db81d61c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:35:08.313 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:08.312+0000 7f176b93f780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:35:10.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:35:10.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:35:10.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:35:10.708 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:35:10.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:35:10.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:35:10.961 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:35:10.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2026-02-22T22:35:10.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:35:10.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:35:10.962 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:35:10.978 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:10.976+0000 7f8ad100f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:10.979 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:10.977+0000 7f8ad100f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:10.980 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:10.978+0000 7f8ad100f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:11.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:35:11.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:35:11.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:35:11.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:35:11.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:35:11.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:11.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:35:11.150 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:11.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:11.150 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:11.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:12.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:12.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:12.339 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:35:12.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:35:12.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:12.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:12.529 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:12.567 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:12.564+0000 7f8ad100f780 -1 Falling back to public interface 2026-02-22T22:35:13.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:13.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:13.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:35:13.531 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:35:13.531 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:13.531 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:13.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:14.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:14.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:14.719 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:35:14.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:35:14.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:14.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:14.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:15.514 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:15.511+0000 7f8ad100f780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:35:15.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:15.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:15.911 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:35:15.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:35:15.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:15.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:16.099 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/572763809,v1:127.0.0.1:6803/572763809] [v2:127.0.0.1:6804/572763809,v1:127.0.0.1:6805/572763809] exists,up 970c8b92-195a-4d1e-8372-a088a7c5c01c 2026-02-22T22:35:16.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:35:16.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:35:16.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: test_get_op_scheduler: get_op_scheduler 0 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: get_op_scheduler: local id=0 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: get_op_scheduler: get_config osd 0 osd_op_queue 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:16.102 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:16.103 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:16.103 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:35:16.103 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:16.103 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_op_queue 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: test_get_op_scheduler: test wpq = wpq 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2333: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:35:16.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:16.153 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:35:16.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:35:16.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:35:16.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-22T22:35:16.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:35:16.155 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:35:16.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab0a5331-d052-48b1-bb62-69389feaf6e7 2026-02-22T22:35:16.155 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 ab0a5331-d052-48b1-bb62-69389feaf6e7 2026-02-22T22:35:16.156 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ab0a5331-d052-48b1-bb62-69389feaf6e7' 2026-02-22T22:35:16.156 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:35:16.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkhJtpQNmzCRAA6XMdlsSvHAj8+nP3Auu7+w== 2026-02-22T22:35:16.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkhJtpQNmzCRAA6XMdlsSvHAj8+nP3Auu7+w=="}' 2026-02-22T22:35:16.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab0a5331-d052-48b1-bb62-69389feaf6e7 -i td/ceph-helpers/1/new.json 2026-02-22T22:35:16.355 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:35:16.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:35:16.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCkhJtpQNmzCRAA6XMdlsSvHAj8+nP3Auu7+w== --osd-uuid ab0a5331-d052-48b1-bb62-69389feaf6e7 2026-02-22T22:35:16.381 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:16.378+0000 7f8380c11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:16.382 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:16.379+0000 7f8380c11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:16.383 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:16.380+0000 7f8380c11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:16.383 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:16.381+0000 7f8380c11780 -1 bdev(0x5591bb1d5c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:35:16.383 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:16.381+0000 7f8380c11780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:35:18.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:35:18.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:35:18.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:35:18.750 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:35:18.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:35:19.003 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:35:19.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:35:19.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-22T22:35:19.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:35:19.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:35:19.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:35:19.021 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:19.018+0000 7f6f1300f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:19.022 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:19.019+0000 7f6f1300f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:19.023 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:19.020+0000 7f6f1300f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:19.192 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:35:19.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:35:19.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:35:19.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:35:19.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:35:19.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:19.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:35:19.193 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:19.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:19.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:19.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:20.355 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:20.353+0000 7f6f1300f780 -1 Falling back to public interface 2026-02-22T22:35:20.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:20.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:20.380 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:35:20.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:35:20.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:20.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:20.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:21.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:21.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:21.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:35:21.573 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:35:21.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:21.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:21.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:22.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:22.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:22.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:35:22.766 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:35:22.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:22.767 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:22.956 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:23.066 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:23.064+0000 7f6f1300f780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:35:23.957 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:23.958 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:35:23.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:23.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:35:23.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:23.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:24.151 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2400397359,v1:127.0.0.1:6811/2400397359] [v2:127.0.0.1:6812/2400397359,v1:127.0.0.1:6813/2400397359] exists,up ab0a5331-d052-48b1-bb62-69389feaf6e7 2026-02-22T22:35:24.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:35:24.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:35:24.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:35:24.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: test_get_op_scheduler: get_op_scheduler 1 2026-02-22T22:35:24.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: get_op_scheduler: local id=1 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: get_op_scheduler: get_config osd 1 osd_op_queue 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.154 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.155 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.1.asok 2026-02-22T22:35:24.155 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:24.155 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-osd.1.asok config get osd_op_queue 2026-02-22T22:35:24.203 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2026-02-22T22:35:24.203 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: test_get_op_scheduler: teardown td/ceph-helpers 2026-02-22T22:35:24.203 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:35:24.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:35:24.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:35:24.204 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:35:24.204 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:35:24.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:35:24.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:35:24.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:35:24.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:35:24.310 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:35:24.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:35:24.311 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:35:24.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:35:24.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:35:24.312 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:35:24.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:24.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:35:24.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:35:24.313 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:24.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:35:24.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:35:24.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:35:24.319 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:35:24.319 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.319 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:35:24.321 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:35:24.321 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:35:24.321 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:35:24.321 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:35:24.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:35:24.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:35:24.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:35:24.323 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:35:24.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:35:24.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:35:24.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:35:24.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:35:24.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:35:24.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:24.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:35:24.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:35:24.326 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:24.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:35:24.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:35:24.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:35:24.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:35:24.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:35:24.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:35:24.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:35:24.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:35:24.330 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:35:24.330 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.330 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:35:24.331 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:35:24.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:35:24.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:24.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:35:24.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:35:24.381 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:35:24.381 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:35:24.381 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:35:24.381 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:35:24.381 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:35:24.382 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:35:24.382 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:35:24.382 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:35:24.382 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:24.382 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.382 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.382 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:35:24.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:24.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:35:24.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:35:24.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:35:24.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:35:24.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:35:24.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:35:24.430 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:35:24.430 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:35:24.430 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:35:24.431 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:24.431 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.431 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.431 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:35:24.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:24.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:35:24.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2026-02-22T22:35:24.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:35:24.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:35:24.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:35:24.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:35:24.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:35:24.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:35:24.576 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:35:24.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:24.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:24.577 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:24.577 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.577 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:24.578 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:35:24.578 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:35:24.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2026-02-22T22:35:24.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:35:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:35:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57932b52-779f-4703-9c32-fc2df374dc5a 2026-02-22T22:35:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 57932b52-779f-4703-9c32-fc2df374dc5a' 2026-02-22T22:35:24.601 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 57932b52-779f-4703-9c32-fc2df374dc5a 2026-02-22T22:35:24.601 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:35:24.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCshJtpR75AJBAAytBBeSjFI9LNVw7ysCSRhg== 2026-02-22T22:35:24.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCshJtpR75AJBAAytBBeSjFI9LNVw7ysCSRhg=="}' 2026-02-22T22:35:24.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57932b52-779f-4703-9c32-fc2df374dc5a -i td/ceph-helpers/0/new.json 2026-02-22T22:35:24.713 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:24.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:35:24.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCshJtpR75AJBAAytBBeSjFI9LNVw7ysCSRhg== --osd-uuid 57932b52-779f-4703-9c32-fc2df374dc5a 2026-02-22T22:35:24.737 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:24.736+0000 7f32528ba780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:24.739 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:24.737+0000 7f32528ba780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:24.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:24.738+0000 7f32528ba780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:24.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:24.738+0000 7f32528ba780 -1 bdev(0x561b3e5a1c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:35:24.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:24.738+0000 7f32528ba780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:35:27.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:35:27.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:35:27.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:35:27.630 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:35:27.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:35:27.885 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:35:27.886 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:35:27.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:35:27.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:35:27.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:35:27.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:35:27.903 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:27.901+0000 7f28fa00c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:27.904 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:27.903+0000 7f28fa00c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:27.905 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:27.904+0000 7f28fa00c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:28.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:35:28.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:35:28.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:35:28.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:35:28.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:35:28.076 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:28.076 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:35:28.076 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:28.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:28.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:28.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:29.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:29.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:29.265 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:35:29.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:35:29.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:29.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:29.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:29.489 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:29.487+0000 7f28fa00c780 -1 Falling back to public interface 2026-02-22T22:35:30.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:30.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:30.458 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:35:30.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:35:30.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:30.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:30.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:31.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:31.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:31.648 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:35:31.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:35:31.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:31.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:31.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:32.433 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:32.431+0000 7f28fa00c780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:35:32.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:32.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:32.839 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:35:32.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:35:32.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:32.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:33.032 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:34.033 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:34.033 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:34.034 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:35:34.034 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:35:34.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:34.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:34.224 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1363583284,v1:127.0.0.1:6803/1363583284] [v2:127.0.0.1:6804/1363583284,v1:127.0.0.1:6805/1363583284] exists,up 57932b52-779f-4703-9c32-fc2df374dc5a 2026-02-22T22:35:34.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:35:34.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:35:34.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:35:34.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2026-02-22T22:35:34.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:35:34.409 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:35:34.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:35:34.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:35:34.652 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:35:34.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:35:35.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:35:35.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2026-02-22T22:35:35.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:35:35.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:35:35.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:35:35.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:35:35.942 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:35:35.942 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:35:35.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:35:35.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:35:35.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:35:36.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:35:36.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:35:36.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:35:36.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:35:36.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:35:36.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:35:36.216 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:35:36.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:35:36.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:35:36.217 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:35:36.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:35:36.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:35:36.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:35:36.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:35:36.279 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:35:36.279 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:35:36.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:35:36.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:35:36.280 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:35:36.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:35:36.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:35:36.281 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:35:36.282 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:36.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:35:36.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:35:37.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:35:37.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:37.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:35:37.661 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:35:37.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:35:37.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:35:37.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:35:38.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:35:38.097 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:35:38.097 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:35:38.097 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:35:38.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:35:38.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:35:38.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:35:38.354 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2026-02-22T22:35:38.354 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2026-02-22T22:35:38.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2026-02-22T22:35:38.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2026-02-22T22:35:38.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2026-02-22T22:35:38.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2026-02-22T22:35:38.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2026-02-22T22:35:38.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2026-02-22T22:35:38.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:35:38.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:35:38.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:35:38.542 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:35:38.542 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:35:38.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:35:38.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:35:38.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:35:38.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:35:38.648 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:35:38.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:35:38.649 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:35:38.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:35:38.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:35:38.650 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:35:38.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:38.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:35:38.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:35:38.652 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:38.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:35:38.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:35:38.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:35:38.657 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:35:38.657 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.657 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_osds td/ceph-helpers 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:35:38.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:35:38.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:35:38.661 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:35:38.661 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:35:38.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:35:38.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:35:38.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:35:38.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:35:38.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:35:38.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:35:38.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:35:38.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:35:38.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:35:38.663 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:35:38.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:38.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:35:38.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:35:38.664 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:35:38.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:35:38.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:35:38.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:35:38.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:35:38.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:35:38.668 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:35:38.668 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:35:38.668 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:35:38.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:35:38.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:35:38.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:35:38.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:35:38.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-22T22:35:38.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:35:38.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:38.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:38.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:38.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.692 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:38.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T22:35:38.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:35:38.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:38.719 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.720 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.720 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:35:38.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:38.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:35:38.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.764 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.765 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:35:38.765 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:35:38.765 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:35:38.810 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2026-02-22T22:35:38.810 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:35:38.810 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:35:38.810 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:35:38.811 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:35:38.811 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:35:38.811 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:35:38.908 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:35:38.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:38.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:38.909 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:38.909 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.909 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.909 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:38.910 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:35:38.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:35:38.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2026-02-22T22:35:38.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:35:38.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:35:38.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:35:38.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a006a61-2897-45fe-9d12-9176b4f0023c 2026-02-22T22:35:38.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5a006a61-2897-45fe-9d12-9176b4f0023c' 2026-02-22T22:35:38.934 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 5a006a61-2897-45fe-9d12-9176b4f0023c 2026-02-22T22:35:38.934 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:35:38.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6hJtp7jkHOBAAOQ8GEl7QFMVcP81ZsT2SSQ== 2026-02-22T22:35:38.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6hJtp7jkHOBAAOQ8GEl7QFMVcP81ZsT2SSQ=="}' 2026-02-22T22:35:38.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a006a61-2897-45fe-9d12-9176b4f0023c -i td/ceph-helpers/0/new.json 2026-02-22T22:35:39.043 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:39.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:35:39.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6hJtp7jkHOBAAOQ8GEl7QFMVcP81ZsT2SSQ== --osd-uuid 5a006a61-2897-45fe-9d12-9176b4f0023c 2026-02-22T22:35:39.067 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:39.066+0000 7f46f040d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:39.069 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:39.067+0000 7f46f040d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:39.070 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:39.068+0000 7f46f040d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:39.070 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:39.068+0000 7f46f040d780 -1 bdev(0x564a191cfc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:35:39.070 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:39.068+0000 7f46f040d780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:35:41.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:35:41.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:35:41.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:35:41.720 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:35:41.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:35:41.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:35:41.973 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:35:41.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:35:41.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:35:41.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:35:41.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:35:41.990 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:41.988+0000 7ff6155b4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:41.991 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:41.990+0000 7ff6155b4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:41.992 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:41.991+0000 7ff6155b4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:42.163 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:35:42.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:35:42.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:35:42.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:35:42.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:35:42.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:42.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:35:42.165 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:42.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:42.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:42.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:43.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:43.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:43.352 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:35:43.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:35:43.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:43.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:43.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:43.585 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:43.584+0000 7ff6155b4780 -1 Falling back to public interface 2026-02-22T22:35:44.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:44.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:44.541 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:35:44.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:35:44.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:44.542 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:44.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:45.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:45.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:45.732 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:35:45.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:35:45.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:45.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:45.923 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:46.294 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:46.291+0000 7ff6155b4780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:35:46.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:46.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:46.925 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:35:46.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:35:46.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:46.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:35:47.115 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1490537282,v1:127.0.0.1:6803/1490537282] [v2:127.0.0.1:6804/1490537282,v1:127.0.0.1:6805/1490537282] exists,up 5a006a61-2897-45fe-9d12-9176b4f0023c 2026-02-22T22:35:47.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:35:47.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:35:47.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:35:47.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2026-02-22T22:35:47.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:35:47.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:35:47.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:35:47.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=475d770e-49bf-46b9-84d1-4e7cd7da9b9f 2026-02-22T22:35:47.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 475d770e-49bf-46b9-84d1-4e7cd7da9b9f' 2026-02-22T22:35:47.123 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 475d770e-49bf-46b9-84d1-4e7cd7da9b9f 2026-02-22T22:35:47.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:35:47.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDhJtpYGuzBxAAGsiAqIMSGVRqNuzX2o0eUw== 2026-02-22T22:35:47.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDhJtpYGuzBxAAGsiAqIMSGVRqNuzX2o0eUw=="}' 2026-02-22T22:35:47.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 475d770e-49bf-46b9-84d1-4e7cd7da9b9f -i td/ceph-helpers/1/new.json 2026-02-22T22:35:47.327 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:35:47.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:35:47.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDhJtpYGuzBxAAGsiAqIMSGVRqNuzX2o0eUw== --osd-uuid 475d770e-49bf-46b9-84d1-4e7cd7da9b9f 2026-02-22T22:35:47.353 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:47.350+0000 7f8aca362780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:47.354 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:47.351+0000 7f8aca362780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:47.355 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:47.352+0000 7f8aca362780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:47.355 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:47.352+0000 7f8aca362780 -1 bdev(0x562b6767bc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:35:47.355 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:47.352+0000 7f8aca362780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:35:49.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:35:49.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:35:49.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:35:49.472 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:35:49.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:35:49.746 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:35:49.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:35:49.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:35:49.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:35:49.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:35:49.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:35:49.762 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:49.760+0000 7f81bc81a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:49.764 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:49.761+0000 7f81bc81a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:49.764 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:49.762+0000 7f81bc81a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:35:49.939 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:35:49.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:35:49.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:35:49.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:35:49.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:35:49.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:49.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:35:49.940 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:35:49.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:49.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:50.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:51.099 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:51.096+0000 7f81bc81a780 -1 Falling back to public interface 2026-02-22T22:35:51.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:51.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:51.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:35:51.129 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:35:51.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:51.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:51.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:52.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:52.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:52.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:35:52.315 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:35:52.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:52.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:52.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:53.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:53.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:53.509 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:35:53.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:35:53.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:53.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:53.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:35:53.787 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:35:53.784+0000 7f81bc81a780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:35:54.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:35:54.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:35:54.698 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:35:54.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:35:54.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:35:54.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:35:54.896 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/976786263,v1:127.0.0.1:6811/976786263] [v2:127.0.0.1:6812/976786263,v1:127.0.0.1:6813/976786263] exists,up 475d770e-49bf-46b9-84d1-4e7cd7da9b9f 2026-02-22T22:35:54.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:35:54.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:35:54.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:35:54.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2026-02-22T22:35:54.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:35:55.083 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:35:55.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:35:55.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:35:55.329 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:35:55.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:35:56.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:35:56.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2026-02-22T22:35:56.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:35:56.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:35:56.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:35:56.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:35:56.618 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:35:56.618 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:35:56.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:35:56.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:35:56.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:35:56.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:35:56.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:35:56.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:35:56.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:35:56.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:35:56.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:35:56.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:35:56.890 INFO:tasks.workunit.client.0.trial123.stderr:1' 2026-02-22T22:35:56.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:35:56.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:35:56.890 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:35:56.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836484 2026-02-22T22:35:56.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836484 2026-02-22T22:35:56.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T22:35:56.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:35:56.956 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:35:57.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672963 2026-02-22T22:35:57.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672963 2026-02-22T22:35:57.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-22T22:35:57.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:35:57.022 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836484 2026-02-22T22:35:57.022 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:35:57.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:35:57.023 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836484 2026-02-22T22:35:57.023 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:35:57.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836484 2026-02-22T22:35:57.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T22:35:57.025 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836484 2026-02-22T22:35:57.025 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:57.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:35:57.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:35:58.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:35:58.212 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:58.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:35:58.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:35:59.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:35:59.405 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:35:59.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836485 -lt 21474836484 2026-02-22T22:35:59.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:35:59.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672963 2026-02-22T22:35:59.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:35:59.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:35:59.598 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672963 2026-02-22T22:35:59.598 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:35:59.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672963 2026-02-22T22:35:59.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T22:35:59.599 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672963 2026-02-22T22:35:59.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:35:59.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-22T22:35:59.787 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:35:59.788 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:35:59.788 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:36:00.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:36:00.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:36:00.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:36:00.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:00.226 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:00.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:36:00.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:36:00.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:36:00.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2026-02-22T22:36:00.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:36:00.694 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:36:00.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:36:00.701 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:36:00.898 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:36:00.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:36:01.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:36:02.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2026-02-22T22:36:02.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2026-02-22T22:36:02.190 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2026-02-22T22:36:02.190 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2026-02-22T22:36:02.190 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2026-02-22T22:36:02.190 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:36:02.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:0' 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:02.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:02.387 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:02.491 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:02.492 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:02.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:02.493 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:02.494 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:02.494 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:02.495 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:02.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:02.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:02.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:02.496 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:02.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:02.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:02.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:02.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:02.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.504 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:02.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:02.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:02.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:36:02.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_pg td/ceph-helpers 2026-02-22T22:36:02.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2026-02-22T22:36:02.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2026-02-22T22:36:02.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:02.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:02.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:02.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:02.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:02.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:02.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:02.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:02.509 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:02.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:02.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:02.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:02.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:02.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:02.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:02.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:02.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:02.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:02.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:02.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:02.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:36:02.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:36:02.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:36:02.516 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:36:02.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:36:02.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:02.538 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:36:02.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:02.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:02.539 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:02.539 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.539 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:02.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:02.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:36:02.566 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:36:02.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:02.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:02.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:36:02.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:36:02.567 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:02.567 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:02.567 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:02.568 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:02.568 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.568 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.569 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:02.569 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:02.569 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:36:02.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:36:02.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:02.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:02.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:36:02.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:36:02.613 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:02.613 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:02.614 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:02.614 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:02.614 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.614 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.614 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:02.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:02.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:36:02.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2026-02-22T22:36:02.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:36:02.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:36:02.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:36:02.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:36:02.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:36:02.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:36:02.758 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:36:02.758 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:02.758 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:02.758 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:02.758 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.759 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.759 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:02.759 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:02.759 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:02.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2026-02-22T22:36:02.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:36:02.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:36:02.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:36:02.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:36:02.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:02.780 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:36:02.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:36:02.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:36:02.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:36:02.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:36:02.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:36:02.782 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:36:02.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd879117-a61d-40a8-8717-271ee8234924 2026-02-22T22:36:02.782 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 dd879117-a61d-40a8-8717-271ee8234924 2026-02-22T22:36:02.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dd879117-a61d-40a8-8717-271ee8234924' 2026-02-22T22:36:02.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:36:02.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDShJtp0A4aLxAAAaOdHZ4R5JI6MSF6lJy1yA== 2026-02-22T22:36:02.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDShJtp0A4aLxAAAaOdHZ4R5JI6MSF6lJy1yA=="}' 2026-02-22T22:36:02.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd879117-a61d-40a8-8717-271ee8234924 -i td/ceph-helpers/0/new.json 2026-02-22T22:36:02.894 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:02.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:36:02.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDShJtp0A4aLxAAAaOdHZ4R5JI6MSF6lJy1yA== --osd-uuid dd879117-a61d-40a8-8717-271ee8234924 2026-02-22T22:36:02.919 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:02.917+0000 7f5d7ce0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:02.921 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:02.919+0000 7f5d7ce0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:02.922 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:02.920+0000 7f5d7ce0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:02.922 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:02.920+0000 7f5d7ce0f780 -1 bdev(0x55e12ff55c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:36:02.922 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:02.920+0000 7f5d7ce0f780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:36:05.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:36:05.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:36:05.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:36:05.314 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:36:05.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:36:05.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:36:05.570 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:36:05.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:36:05.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:36:05.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:36:05.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:36:05.587 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:05.585+0000 7f256d615780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:05.588 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:05.586+0000 7f256d615780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:05.589 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:05.587+0000 7f256d615780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:05.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:36:05.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:36:05.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:36:05.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:36:05.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:36:05.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:05.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:36:05.758 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:05.759 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:05.759 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:05.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:06.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:06.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:06.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:36:06.945 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:36:06.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:06.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:07.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:07.171 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:07.169+0000 7f256d615780 -1 Falling back to public interface 2026-02-22T22:36:08.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:08.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:08.136 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:36:08.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:36:08.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:08.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:08.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:09.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:09.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:09.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:36:09.326 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:36:09.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:09.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:09.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:09.863 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:09.862+0000 7f256d615780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:36:10.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:10.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:10.517 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:36:10.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:36:10.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:10.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:10.708 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/92189671,v1:127.0.0.1:6803/92189671] [v2:127.0.0.1:6804/92189671,v1:127.0.0.1:6805/92189671] exists,up dd879117-a61d-40a8-8717-271ee8234924 2026-02-22T22:36:10.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:36:10.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:36:10.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:36:10.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2026-02-22T22:36:10.709 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:36:10.894 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:36:10.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:36:10.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:36:11.172 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:36:11.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:36:12.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:36:12.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2026-02-22T22:36:12.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:36:12.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:36:12.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:36:12.463 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:36:12.464 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:12.464 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:12.464 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:12.464 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:12.464 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:12.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:36:12.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:36:12.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:36:12.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:36:12.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:36:12.546 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:36:12.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:36:12.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:36:12.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:36:12.733 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:36:12.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:36:12.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:36:12.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:36:12.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:36:12.798 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:36:12.798 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:36:12.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:36:12.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:36:12.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:36:12.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:36:12.800 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:36:12.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:36:12.801 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:12.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:36:12.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:36:13.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:36:13.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:14.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:36:14.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:36:15.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:36:15.176 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:15.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:36:15.369 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:36:15.369 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:15.369 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:15.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:36:15.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:36:15.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:36:15.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:36:15.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:36:15.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:36:15.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:36:15.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:36:15.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:36:15.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:36:15.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:15.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:16.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:36:16.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:36:16.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:36:16.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2026-02-22T22:36:16.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2026-02-22T22:36:16.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2026-02-22T22:36:16.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2026-02-22T22:36:16.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2026-02-22T22:36:16.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:16.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:16.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:16.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:16.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:16.369 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:16.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:16.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:16.371 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:16.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:16.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:16.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:16.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:16.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:16.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:16.375 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:16.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:16.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:16.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:16.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:16.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_primary td/ceph-helpers 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:16.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:16.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:16.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:16.382 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:16.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:16.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:16.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:16.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:16.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:16.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:16.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:16.385 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:16.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:16.386 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:16.387 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:16.387 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:16.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:16.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:16.389 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:16.389 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:16.389 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:36:16.390 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:36:16.390 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.390 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:36:16.391 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:36:16.392 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:36:16.392 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:36:16.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:16.412 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:36:16.412 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:16.412 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:16.413 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:16.413 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.413 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.413 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:16.413 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:16.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:36:16.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:36:16.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:16.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:16.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:36:16.440 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:36:16.440 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:16.440 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:16.440 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:16.441 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:16.441 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.441 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.441 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:16.441 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:16.441 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:36:16.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:36:16.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:16.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:16.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:36:16.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:36:16.486 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:16.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:36:16.529 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2026-02-22T22:36:16.529 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2026-02-22T22:36:16.529 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:36:16.529 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:36:16.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:36:16.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:36:16.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:36:16.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:36:16.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:36:16.626 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:16.626 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:16.626 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:16.626 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.626 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.626 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:16.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:16.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:36:16.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:16.648 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:36:16.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:36:16.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:36:16.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:36:16.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:36:16.650 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:36:16.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=135b20d2-ebd8-4355-bc87-6d5501aeb5fa 2026-02-22T22:36:16.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 135b20d2-ebd8-4355-bc87-6d5501aeb5fa' 2026-02-22T22:36:16.650 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 135b20d2-ebd8-4355-bc87-6d5501aeb5fa 2026-02-22T22:36:16.651 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:36:16.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDghJtpF7koJxAAamjK3J5+Det8XkO4MssiMA== 2026-02-22T22:36:16.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDghJtpF7koJxAAamjK3J5+Det8XkO4MssiMA=="}' 2026-02-22T22:36:16.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 135b20d2-ebd8-4355-bc87-6d5501aeb5fa -i td/ceph-helpers/0/new.json 2026-02-22T22:36:16.764 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:36:16.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDghJtpF7koJxAAamjK3J5+Det8XkO4MssiMA== --osd-uuid 135b20d2-ebd8-4355-bc87-6d5501aeb5fa 2026-02-22T22:36:16.789 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:16.787+0000 7fdffb63f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:16.790 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:16.788+0000 7fdffb63f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:16.791 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:16.789+0000 7fdffb63f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:16.791 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:16.790+0000 7fdffb63f780 -1 bdev(0x55a22c7a7c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:36:16.791 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:16.790+0000 7fdffb63f780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:36:19.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:36:19.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:36:19.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:36:19.442 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:36:19.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:36:19.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:36:19.697 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:36:19.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:36:19.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:36:19.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:36:19.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:36:19.712 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:19.711+0000 7f39cad64780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:19.714 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:19.712+0000 7f39cad64780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:19.715 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:19.713+0000 7f39cad64780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:19.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:36:19.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:36:19.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:36:19.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:36:19.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:36:19.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:19.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:36:19.893 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:19.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:19.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:20.078 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:21.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:21.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:21.080 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:36:21.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:36:21.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:21.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:21.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:21.299 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:21.297+0000 7f39cad64780 -1 Falling back to public interface 2026-02-22T22:36:22.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:22.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:22.269 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:36:22.269 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:36:22.269 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:22.270 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:22.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:23.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:23.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:23.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:36:23.458 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:36:23.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:23.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:23.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:24.011 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:24.009+0000 7f39cad64780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:36:24.649 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:24.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:24.650 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:36:24.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:36:24.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:24.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:24.846 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3901046947,v1:127.0.0.1:6803/3901046947] [v2:127.0.0.1:6804/3901046947,v1:127.0.0.1:6805/3901046947] exists,up 135b20d2-ebd8-4355-bc87-6d5501aeb5fa 2026-02-22T22:36:24.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:36:24.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:36:24.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:36:24.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2026-02-22T22:36:24.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:36:25.030 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:36:25.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:36:25.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:36:25.319 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:36:25.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:36:26.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:36:26.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2026-02-22T22:36:26.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:36:26.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:36:26.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:36:26.610 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:36:26.610 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:26.610 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:26.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:26.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:26.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:26.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:36:26.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:36:26.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:36:26.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:36:26.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:36:26.693 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:36:26.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:36:26.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:36:26.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:36:26.881 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:36:26.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:36:26.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:36:26.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:36:26.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:36:26.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:36:26.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:36:26.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:36:26.948 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:36:26.948 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:36:26.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:36:26.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:36:26.949 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:36:26.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:27.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:36:27.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:36:28.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:36:28.140 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:28.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:36:28.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:36:29.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:36:29.329 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:29.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:36:29.520 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:36:29.520 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:29.520 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:29.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:36:29.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:36:29.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:36:29.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:36:29.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:36:29.776 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:36:29.776 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:36:29.776 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:36:29.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:36:29.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:36:29.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:29.962 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:30.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:36:30.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:36:30.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:36:30.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2026-02-22T22:36:30.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2026-02-22T22:36:30.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2026-02-22T22:36:30.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2026-02-22T22:36:30.216 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2026-02-22T22:36:30.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:30.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:30.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:30.512 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:30.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:30.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:30.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:30.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:30.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:30.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:30.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:30.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:30.517 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:30.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:30.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:30.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:30.522 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:30.522 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:30.522 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:30.522 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:30.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:30.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:30.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:36:30.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_get_timeout_delays td/ceph-helpers 2026-02-22T22:36:30.526 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: get_timeout_delays 1 2026-02-22T22:36:30.526 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.526 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.526 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.526 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.527 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: test '1 ' = '1 ' 2026-02-22T22:36:30.535 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: get_timeout_delays 5 2026-02-22T22:36:30.535 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.536 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2026-02-22T22:36:30.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: get_timeout_delays 6 2026-02-22T22:36:30.554 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.554 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.554 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.554 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.554 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2026-02-22T22:36:30.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: get_timeout_delays 7 2026-02-22T22:36:30.572 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.572 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2026-02-22T22:36:30.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: test_get_timeout_delays: get_timeout_delays 8 2026-02-22T22:36:30.595 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.595 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2026-02-22T22:36:30.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: test_get_timeout_delays: get_timeout_delays 1 .1 2026-02-22T22:36:30.620 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.620 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.644 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2026-02-22T22:36:30.644 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: test_get_timeout_delays: get_timeout_delays 1.5 .1 2026-02-22T22:36:30.645 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.645 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.645 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.646 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.646 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2026-02-22T22:36:30.673 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: test_get_timeout_delays: get_timeout_delays 5 .1 2026-02-22T22:36:30.674 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.674 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.712 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2026-02-22T22:36:30.712 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: test_get_timeout_delays: get_timeout_delays 6 .1 2026-02-22T22:36:30.713 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.713 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.713 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.713 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.713 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2026-02-22T22:36:30.750 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: test_get_timeout_delays: get_timeout_delays 6.3 .1 2026-02-22T22:36:30.751 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.751 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.751 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.752 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.752 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2026-02-22T22:36:30.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: test_get_timeout_delays: get_timeout_delays 20 .1 2026-02-22T22:36:30.787 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.787 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.787 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.788 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.788 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2026-02-22T22:36:30.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: test_get_timeout_delays: get_timeout_delays 300 .1 0 2026-02-22T22:36:30.836 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.836 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:30.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2026-02-22T22:36:30.910 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: test_get_timeout_delays: get_timeout_delays 300 .1 10 2026-02-22T22:36:30.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:30.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:30.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:30.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:30.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2026-02-22T22:36:31.136 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_is_clean td/ceph-helpers 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:31.137 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:31.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:31.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:31.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:31.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:31.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:31.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:31.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:31.140 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:31.140 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:31.140 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:31.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:31.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:31.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:31.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:31.142 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:31.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:31.143 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:31.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:31.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:31.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:31.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:31.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:31.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:31.145 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:36:31.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:36:31.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:31.146 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:31.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:36:31.147 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:36:31.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:36:31.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:36:31.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:36:31.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:31.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:36:31.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:36:31.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:36:31.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:36:31.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:36:31.149 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:31.168 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:36:31.168 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:31.168 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:31.168 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:31.168 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:31.168 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:31.168 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:31.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:31.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:31.195 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:31.196 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:31.196 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:31.196 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:31.196 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:36:31.239 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:36:31.239 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:31.239 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:31.240 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:31.241 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:31.241 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:36:31.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2026-02-22T22:36:31.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:36:31.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:36:31.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:36:31.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:36:31.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:36:31.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:36:31.387 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:36:31.387 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:31.387 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:31.387 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:31.388 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:31.388 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:31.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:31.388 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:31.389 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:36:31.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:31.409 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:36:31.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:36:31.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:36:31.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:36:31.411 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:36:31.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=879209ea-0600-40a0-bcac-177d2e78d0de 2026-02-22T22:36:31.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 879209ea-0600-40a0-bcac-177d2e78d0de' 2026-02-22T22:36:31.411 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 879209ea-0600-40a0-bcac-177d2e78d0de 2026-02-22T22:36:31.411 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:36:31.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvhJtpW4/gGBAA4D+hFuZajBt77JW1nm6ypA== 2026-02-22T22:36:31.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvhJtpW4/gGBAA4D+hFuZajBt77JW1nm6ypA=="}' 2026-02-22T22:36:31.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 879209ea-0600-40a0-bcac-177d2e78d0de -i td/ceph-helpers/0/new.json 2026-02-22T22:36:31.521 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:31.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:36:31.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvhJtpW4/gGBAA4D+hFuZajBt77JW1nm6ypA== --osd-uuid 879209ea-0600-40a0-bcac-177d2e78d0de 2026-02-22T22:36:31.546 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:31.543+0000 7f6a85a0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:31.547 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:31.544+0000 7f6a85a0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:31.548 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:31.545+0000 7f6a85a0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:31.548 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:31.545+0000 7f6a85a0c780 -1 bdev(0x56135c69fc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:36:31.548 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:31.546+0000 7f6a85a0c780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:36:34.446 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:36:34.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:36:34.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:36:34.448 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:36:34.448 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:36:34.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:36:34.709 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:36:34.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:36:34.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:36:34.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:36:34.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:36:34.725 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:34.724+0000 7f6abd023780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:34.727 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:34.725+0000 7f6abd023780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:34.728 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:34.726+0000 7f6abd023780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:34.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:36:34.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:36:34.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:36:34.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:36:34.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:36:34.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:34.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:36:34.896 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:34.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:34.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:35.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:36.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:36.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:36.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:36:36.086 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:36:36.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:36.087 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:36.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:36.307 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:36.305+0000 7f6abd023780 -1 Falling back to public interface 2026-02-22T22:36:37.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:37.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:37.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:36:37.278 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:36:37.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:37.279 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:37.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:38.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:38.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:38.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:36:38.475 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:36:38.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:38.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:38.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:39.013 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:39.011+0000 7f6abd023780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:36:39.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:39.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:39.670 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:36:39.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:36:39.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:39.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:39.860 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2220018176,v1:127.0.0.1:6803/2220018176] [v2:127.0.0.1:6804/2220018176,v1:127.0.0.1:6805/2220018176] exists,up 879209ea-0600-40a0-bcac-177d2e78d0de 2026-02-22T22:36:39.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:36:39.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:36:39.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:36:39.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2026-02-22T22:36:39.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:36:40.043 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:36:40.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:36:40.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:36:40.319 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:36:40.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:36:41.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:36:41.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:36:41.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:36:41.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:36:41.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:36:41.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:36:41.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:36:41.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:36:41.693 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:36:41.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:36:41.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:36:41.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:36:41.882 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:36:41.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836484 2026-02-22T22:36:41.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836484 2026-02-22T22:36:41.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T22:36:41.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:36:41.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836484 2026-02-22T22:36:41.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:36:41.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:36:41.948 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836484 2026-02-22T22:36:41.948 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:36:41.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836484 2026-02-22T22:36:41.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T22:36:41.950 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836484 2026-02-22T22:36:41.950 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:42.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:36:42.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:36:43.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:36:43.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:43.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:36:43.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:36:44.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:36:44.330 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:36:44.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T22:36:44.524 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:36:44.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:44.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:44.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:36:44.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:36:44.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:36:44.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:36:44.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:36:44.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:36:44.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:36:44.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:36:44.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:36:44.969 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:36:44.970 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:44.970 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:45.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:36:45.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:36:45.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:36:45.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2026-02-22T22:36:45.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2026-02-22T22:36:45.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:36:45.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:36:45.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:45.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:45.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:45.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:45.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:45.769 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:45.770 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:45.770 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:45.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:45.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:45.772 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:45.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:45.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:45.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:45.774 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:45.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:45.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:45.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:45.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:45.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:45.780 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:45.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_kill_daemon td/ceph-helpers 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:36:45.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:36:45.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:36:45.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:36:45.783 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:36:45.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:36:45.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:36:45.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:36:45.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:36:45.784 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:36:45.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:36:45.784 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:36:45.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:36:45.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:36:45.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:36:45.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:45.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:36:45.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:36:45.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:36:45.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:36:45.788 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:36:45.788 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:36:45.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:36:45.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:45.789 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:45.789 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:36:45.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:36:45.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:36:45.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:36:45.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:36:45.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:45.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:45.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:36:45.792 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:36:45.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:45.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:36:45.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:45.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:45.814 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:45.814 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:45.814 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:45.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:45.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:36:45.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:45.842 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:45.843 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:45.843 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:45.843 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:45.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:45.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:36:45.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:36:45.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:36:45.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:36:45.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:36:45.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:36:45.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:36:45.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2026-02-22T22:36:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:36:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:36:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:36:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:36:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:36:45.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:36:46.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:36:46.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:46.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:46.028 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:46.028 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:46.028 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:46.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:46.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:46.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:36:46.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2026-02-22T22:36:46.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:36:46.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:36:46.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:36:46.050 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:36:46.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1d4da87-a848-4a5b-87a1-059913ab1be4 2026-02-22T22:36:46.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e1d4da87-a848-4a5b-87a1-059913ab1be4' 2026-02-22T22:36:46.052 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 e1d4da87-a848-4a5b-87a1-059913ab1be4 2026-02-22T22:36:46.052 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:36:46.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+hJtpv8OFAxAAy97r48qiTLiYnD2vYtwKKg== 2026-02-22T22:36:46.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+hJtpv8OFAxAAy97r48qiTLiYnD2vYtwKKg=="}' 2026-02-22T22:36:46.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1d4da87-a848-4a5b-87a1-059913ab1be4 -i td/ceph-helpers/0/new.json 2026-02-22T22:36:46.164 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:46.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:36:46.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+hJtpv8OFAxAAy97r48qiTLiYnD2vYtwKKg== --osd-uuid e1d4da87-a848-4a5b-87a1-059913ab1be4 2026-02-22T22:36:46.189 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:46.187+0000 7f64b333f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:46.190 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:46.189+0000 7f64b333f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:46.191 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:46.189+0000 7f64b333f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:46.192 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:46.190+0000 7f64b333f780 -1 bdev(0x557d7a23dc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:36:46.192 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:46.190+0000 7f64b333f780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:36:48.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:36:48.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:36:48.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:36:48.850 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:36:48.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:36:49.104 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:36:49.105 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:36:49.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:36:49.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:36:49.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:36:49.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:36:49.120 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:49.118+0000 7f4533962780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:49.121 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:49.120+0000 7f4533962780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:49.122 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:49.121+0000 7f4533962780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:36:49.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:49.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:49.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:50.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:50.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:50.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:36:50.485 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:36:50.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:50.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:50.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:50.713 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:50.711+0000 7f4533962780 -1 Falling back to public interface 2026-02-22T22:36:51.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:51.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:51.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:36:51.679 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:36:51.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:51.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:51.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:52.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:52.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:52.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:36:52.868 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:36:52.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:52.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:53.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:53.660 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:36:53.659+0000 7f4533962780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:36:54.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:54.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:54.056 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:36:54.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:36:54.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:54.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:54.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:36:55.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:36:55.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:36:55.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:36:55.251 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:36:55.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:36:55.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:36:55.439 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3606056809,v1:127.0.0.1:6803/3606056809] [v2:127.0.0.1:6804/3606056809,v1:127.0.0.1:6805/3606056809] exists,up e1d4da87-a848-4a5b-87a1-059913ab1be4 2026-02-22T22:36:55.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:36:55.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:36:55.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:36:55.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2026-02-22T22:36:55.440 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2026-02-22T22:36:55.441 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2026-02-22T22:36:55.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-22T22:36:55.441 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2026-02-22T22:36:55.441 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=126045 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 126045 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:55.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:55.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2026-02-22T22:36:56.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:56.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 126045 2026-02-22T22:36:56.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:56.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:56.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2026-02-22T22:36:56.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2026-02-22T22:36:56.445 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2026-02-22T22:36:56.445 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=126045 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 126045 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:56.447 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2026-02-22T22:36:56.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:56.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 126045 2026-02-22T22:36:56.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:56.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:56.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2026-02-22T22:36:56.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:56.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 126045 2026-02-22T22:36:56.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:56.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:56.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2026-02-22T22:36:57.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:57.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 126045 2026-02-22T22:36:57.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2026-02-22T22:36:57.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2026-02-22T22:36:57.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2026-02-22T22:36:57.749 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2026-02-22T22:36:57.750 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2026-02-22T22:36:57.750 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2026-02-22T22:36:57.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-22T22:36:57.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2026-02-22T22:36:57.752 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2026-02-22T22:36:57.753 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=125099 2026-02-22T22:36:57.753 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2026-02-22T22:36:57.753 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-22T22:36:57.753 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2026-02-22T22:36:57.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:57.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 125099 2026-02-22T22:36:57.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:57.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:57.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2026-02-22T22:36:57.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:57.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125099 2026-02-22T22:36:57.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2026-02-22T22:36:57.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2026-02-22T22:36:57.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2026-02-22T22:36:57.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2026-02-22T22:36:57.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2026-02-22T22:36:57.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2026-02-22T22:36:57.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-22T22:36:57.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2026-02-22T22:36:57.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=125014 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 125014 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:57.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2026-02-22T22:36:57.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:57.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125014 2026-02-22T22:36:57.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2026-02-22T22:36:57.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2026-02-22T22:36:57.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2026-02-22T22:36:58.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2026-02-22T22:36:58.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125014 2026-02-22T22:36:58.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2026-02-22T22:36:58.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2026-02-22T22:36:58.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2026-02-22T22:36:58.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2026-02-22T22:37:03.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2026-02-22T22:37:03.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:37:03.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:37:03.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:37:03.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:03.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:03.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:03.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:03.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:03.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:03.172 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:37:03.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:37:03.173 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:37:03.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:37:03.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:37:03.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:37:03.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:03.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:37:03.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:37:03.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:03.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:37:03.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:37:03.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:37:03.180 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:37:03.180 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.180 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_kill_daemons td/ceph-helpers 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:37:03.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:37:03.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:03.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:03.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:03.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:03.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:03.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:03.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:37:03.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:37:03.184 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:37:03.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:37:03.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:37:03.185 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:37:03.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:03.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:37:03.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:37:03.187 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:03.187 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:37:03.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:37:03.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:37:03.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:37:03.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:37:03.190 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:37:03.190 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:37:03.190 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:37:03.191 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:37:03.192 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.192 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.192 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:37:03.193 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:37:03.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:37:03.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:03.214 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:37:03.214 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:03.214 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:03.214 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:03.214 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.214 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.215 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:03.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:03.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:37:03.243 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:37:03.243 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:37:03.244 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:37:03.244 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:37:03.244 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:37:03.244 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:37:03.244 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:37:03.244 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:37:03.244 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:37:03.245 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.245 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.245 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:37:03.245 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:37:03.245 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:37:03.288 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:37:03.289 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:37:03.289 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.289 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.289 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:37:03.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:37:03.289 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:37:03.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2026-02-22T22:37:03.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:37:03.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:37:03.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:37:03.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:37:03.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:37:03.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:37:03.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:37:03.435 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:03.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:03.436 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:03.436 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.436 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:03.437 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:37:03.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:37:03.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2026-02-22T22:37:03.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:37:03.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:37:03.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:37:03.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:37:03.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:37:03.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:37:03.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:37:03.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:37:03.459 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:37:03.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:37:03.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:37:03.462 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:37:03.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4e1f6eea-94d8-4365-b95d-53611d7a6740 2026-02-22T22:37:03.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4e1f6eea-94d8-4365-b95d-53611d7a6740' 2026-02-22T22:37:03.462 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 4e1f6eea-94d8-4365-b95d-53611d7a6740 2026-02-22T22:37:03.462 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:37:03.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPhZtp39i7GxAARMarBDQGPJV/tdwGD1pm1w== 2026-02-22T22:37:03.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPhZtp39i7GxAARMarBDQGPJV/tdwGD1pm1w=="}' 2026-02-22T22:37:03.467 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4e1f6eea-94d8-4365-b95d-53611d7a6740 -i td/ceph-helpers/0/new.json 2026-02-22T22:37:03.567 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:37:03.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:37:03.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPhZtp39i7GxAARMarBDQGPJV/tdwGD1pm1w== --osd-uuid 4e1f6eea-94d8-4365-b95d-53611d7a6740 2026-02-22T22:37:03.591 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:03.589+0000 7fbbaa791780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:03.593 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:03.590+0000 7fbbaa791780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:03.594 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:03.591+0000 7fbbaa791780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:03.594 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:03.591+0000 7fbbaa791780 -1 bdev(0x55e4a0997c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:37:03.594 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:03.591+0000 7fbbaa791780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:37:06.246 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:37:06.246 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:37:06.246 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:37:06.247 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:37:06.247 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:37:06.501 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:37:06.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:37:06.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:37:06.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:37:06.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:37:06.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:37:06.518 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:06.516+0000 7fdbd9818780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:06.519 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:06.518+0000 7fdbd9818780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:06.520 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:06.518+0000 7fdbd9818780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:06.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:37:06.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:37:06.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:37:06.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:37:06.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:37:06.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:06.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:37:06.692 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:37:06.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:06.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:06.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:07.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:07.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:07.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:37:07.879 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:37:07.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:07.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:08.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:08.105 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:08.103+0000 7fdbd9818780 -1 Falling back to public interface 2026-02-22T22:37:09.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:09.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:09.072 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:37:09.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:37:09.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:09.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:09.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:10.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:10.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:10.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:37:10.262 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:37:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:10.453 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:11.059 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:11.057+0000 7fdbd9818780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:37:11.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:11.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:11.454 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:37:11.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:37:11.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:11.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:11.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:12.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:12.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:12.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:37:12.653 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:37:12.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:12.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:12.840 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3442650822,v1:127.0.0.1:6803/3442650822] [v2:127.0.0.1:6804/3442650822,v1:127.0.0.1:6805/3442650822] exists,up 4e1f6eea-94d8-4365-b95d-53611d7a6740 2026-02-22T22:37:12.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:37:12.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:37:12.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:37:12.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2026-02-22T22:37:12.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:12.842 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:12.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:12.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:12.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:13.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2026-02-22T22:37:13.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2026-02-22T22:37:13.848 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:13.848 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:13.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:13.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:13.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:15.157 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:15.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2026-02-22T22:37:15.158 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:15.159 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:15.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:15.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:15.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:15.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:15.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2026-02-22T22:37:15.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:15.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:15.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:15.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:15.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:15.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:15.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:20.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:20.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:20.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:20.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:37:20.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:37:20.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:37:20.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:37:20.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:37:20.588 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:37:20.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:20.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:37:20.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:37:20.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:20.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:37:20.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:37:20.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:37:20.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:37:20.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:37:20.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:37:20.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:37:20.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:37:20.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_objectstore_tool td/ceph-helpers 2026-02-22T22:37:20.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2026-02-22T22:37:20.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2026-02-22T22:37:20.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:20.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:20.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:20.597 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:37:20.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:37:20.597 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:37:20.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:37:20.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:37:20.598 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:37:20.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:20.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:37:20.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:37:20.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:20.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:37:20.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:37:20.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:37:20.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:37:20.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:37:20.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:37:20.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:37:20.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:37:20.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:37:20.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:37:20.606 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:37:20.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:37:20.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:37:20.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:37:20.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:20.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:37:20.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:37:20.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:37:20.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:37:20.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:37:20.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:20.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:20.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:37:20.653 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:37:20.653 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:37:20.653 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:37:20.653 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:37:20.654 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:37:20.654 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:37:20.654 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:37:20.654 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:37:20.654 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:37:20.654 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.654 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.655 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:37:20.655 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:37:20.655 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:37:20.699 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:37:20.700 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:37:20.700 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.700 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.700 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:37:20.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:37:20.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:37:20.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:37:20.844 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:37:20.844 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:20.845 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:20.845 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:20.845 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.845 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.845 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:20.845 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:37:20.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:37:20.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2026-02-22T22:37:20.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:37:20.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:37:20.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:37:20.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:20.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:37:20.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f998e652-259f-4283-9fcc-4098ae0be47c 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 f998e652-259f-4283-9fcc-4098ae0be47c 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f998e652-259f-4283-9fcc-4098ae0be47c' 2026-02-22T22:37:20.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:37:20.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAghZtpNYYrNBAALREPEh4bMM1uXniuri6vAg== 2026-02-22T22:37:20.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAghZtpNYYrNBAALREPEh4bMM1uXniuri6vAg=="}' 2026-02-22T22:37:20.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f998e652-259f-4283-9fcc-4098ae0be47c -i td/ceph-helpers/0/new.json 2026-02-22T22:37:20.979 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:37:20.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:37:20.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAghZtpNYYrNBAALREPEh4bMM1uXniuri6vAg== --osd-uuid f998e652-259f-4283-9fcc-4098ae0be47c 2026-02-22T22:37:21.004 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:21.002+0000 7fb1ddd09780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:21.005 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:21.003+0000 7fb1ddd09780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:21.006 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:21.004+0000 7fb1ddd09780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:21.007 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:21.005+0000 7fb1ddd09780 -1 bdev(0x55dea7747c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:37:21.007 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:21.005+0000 7fb1ddd09780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:37:23.654 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:37:23.654 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:37:23.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:37:23.655 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:37:23.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:37:23.911 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:37:23.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:37:23.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:37:23.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:37:23.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:37:23.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:37:23.928 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:23.926+0000 7f2bba01c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:23.929 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:23.927+0000 7f2bba01c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:23.930 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:23.928+0000 7f2bba01c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:24.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:24.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:25.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:25.291 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:37:25.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:37:25.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:25.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:25.479 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:25.522 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:25.520+0000 7f2bba01c780 -1 Falling back to public interface 2026-02-22T22:37:26.480 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:26.480 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:26.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:37:26.481 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:37:26.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:26.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:27.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:27.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:27.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:37:27.674 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:37:27.674 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:27.674 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:27.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:28.460 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:28.459+0000 7f2bba01c780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:37:28.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:28.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:28.869 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:37:28.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:37:28.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:28.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:29.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:30.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:30.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:30.062 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:37:30.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:37:30.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:30.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:30.252 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2404438243,v1:127.0.0.1:6803/2404438243] [v2:127.0.0.1:6804/2404438243,v1:127.0.0.1:6805/2404438243] exists,up f998e652-259f-4283-9fcc-4098ae0be47c 2026-02-22T22:37:30.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:37:30.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:37:30.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:37:30.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2026-02-22T22:37:30.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:37:30.437 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:37:30.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:37:30.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:37:30.682 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:37:30.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:37:31.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:37:31.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2026-02-22T22:37:31.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:37:31.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:37:31.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:37:31.972 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:37:31.972 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:37:31.972 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:37:31.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:37:31.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:37:31.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:37:32.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:37:32.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:37:32.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:37:32.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:37:32.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:37:32.055 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:37:32.242 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:37:32.242 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:37:32.242 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:37:32.243 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:37:32.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:37:32.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:37:32.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:37:32.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:37:32.307 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:37:32.307 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:37:32.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:37:32.308 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:37:32.308 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:37:32.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:37:32.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:37:32.309 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:37:32.310 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:37:32.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:37:32.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:37:33.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:37:33.499 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:37:33.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:37:33.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:37:33.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:37:33.687 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:37:33.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:37:33.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:37:33.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:37:33.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:37:33.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:37:33.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:37:33.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:37:33.942 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:37:34.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:37:34.130 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:37:34.130 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:37:34.130 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:37:34.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:37:34.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:37:34.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:37:34.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2026-02-22T22:37:34.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2026-02-22T22:37:34.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2026-02-22T22:37:34.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:34.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:35.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2026-02-22T22:37:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-22T22:37:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:37:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2026-02-22T22:37:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:37:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-22T22:37:35.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2026-02-22T22:37:36.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2026-02-22T22:37:36.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2026-02-22T22:37:36.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:37:36.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T22:37:36.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:37:36.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:37:36.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:36.285 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:37:36.286 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:start osd.0 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:37:36.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:37:36.298 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:36.295+0000 7ff2a6fa3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:36.299 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:36.296+0000 7ff2a6fa3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:36.300 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:36.297+0000 7ff2a6fa3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:36.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T22:37:36.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:37:36.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:37:36.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:37:36.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:37:36.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:36.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:37:36.473 INFO:tasks.workunit.client.0.trial123.stderr:0 2026-02-22T22:37:36.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:36.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:36.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:37.638 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:37.635+0000 7ff2a6fa3780 -1 Falling back to public interface 2026-02-22T22:37:37.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:37.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:37.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:37:37.661 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:37:37.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:37.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:37.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:38.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:38.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:38.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:37:38.855 INFO:tasks.workunit.client.0.trial123.stderr:2 2026-02-22T22:37:38.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:38.855 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:39.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:40.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:40.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:40.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:37:40.045 INFO:tasks.workunit.client.0.trial123.stderr:3 2026-02-22T22:37:40.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:40.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:40.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:40.611 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:40.609+0000 7ff2a6fa3780 -1 osd.0 12 log_to_monitors true 2026-02-22T22:37:41.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:41.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:41.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:37:41.235 INFO:tasks.workunit.client.0.trial123.stderr:4 2026-02-22T22:37:41.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:41.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:41.426 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 up in weight 1 up_from 16 up_thru 16 down_at 12 last_clean_interval [5,12) [v2:127.0.0.1:6802/3398280114,v1:127.0.0.1:6803/3398280114] [v2:127.0.0.1:6804/3398280114,v1:127.0.0.1:6805/3398280114] exists,up f998e652-259f-4283-9fcc-4098ae0be47c 2026-02-22T22:37:41.426 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:37:41.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:37:41.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:37:41.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:37:41.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:37:41.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:37:41.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:37:41.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:37:41.428 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:37:41.428 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:37:41.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:37:41.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:37:41.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:37:41.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:37:41.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:37:41.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:37:41.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:37:41.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:37:41.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:37:41.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:37:41.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:37:41.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:37:41.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:37:41.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=68719476739 2026-02-22T22:37:41.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 68719476739 2026-02-22T22:37:41.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-68719476739' 2026-02-22T22:37:41.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:37:41.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-68719476739 2026-02-22T22:37:41.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:37:41.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:37:41.765 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-68719476739 2026-02-22T22:37:41.765 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:37:41.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=68719476739 2026-02-22T22:37:41.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2026-02-22T22:37:41.766 INFO:tasks.workunit.client.0.trial123.stderr:waiting osd.0 seq 68719476739 2026-02-22T22:37:41.766 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:37:41.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 1 -lt 68719476739 2026-02-22T22:37:41.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:37:42.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:37:42.954 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:37:43.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 68719476739 -lt 68719476739 2026-02-22T22:37:43.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:37:43.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:37:43.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:37:43.392 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:37:43.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:37:43.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:37:43.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:37:43.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:37:43.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:37:43.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:37:43.393 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:37:43.578 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:37:43.579 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:37:43.579 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:37:43.579 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:37:43.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:37:43.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:37:43.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:37:43.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-22T22:37:43.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2026-02-22T22:37:43.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:43.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:43.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:43.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:45.140 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:45.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-22T22:37:45.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-22T22:37:45.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:37:45.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2026-02-22T22:37:45.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:37:45.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-22T22:37:45.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2026-02-22T22:37:45.443 INFO:tasks.workunit.client.0.trial123.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:37:45.959 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:45.960 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:45.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:45.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:45.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:46.065 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:46.066 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:37:46.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:37:46.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:37:46.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:37:46.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:37:46.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:37:46.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:46.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:37:46.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:37:46.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:46.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:37:46.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:37:46.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:37:46.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:37:46.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.077 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_pg_schedule_scrub td/ceph-helpers 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_pg_schedule_scrub: local dir=td/ceph-helpers 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1998: test_pg_schedule_scrub: setup td/ceph-helpers 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:37:46.079 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:37:46.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:37:46.080 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:37:46.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:37:46.081 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:37:46.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:37:46.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:37:46.083 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:37:46.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:46.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:37:46.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:37:46.084 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:37:46.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:37:46.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:37:46.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:37:46.087 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:37:46.087 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.087 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.087 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:37:46.088 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:37:46.088 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:37:46.088 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:37:46.089 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:37:46.089 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.090 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:37:46.090 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:37:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:37:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1999: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:37:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:46.112 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:37:46.112 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:46.112 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:46.112 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:46.113 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.113 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.113 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:46.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:37:46.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:37:46.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:37:46.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:37:46.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:37:46.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:37:46.141 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:37:46.141 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:37:46.142 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:37:46.142 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:37:46.142 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:37:46.142 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.142 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.142 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:37:46.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:37:46.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:37:46.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:37:46.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:37:46.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:37:46.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:37:46.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:37:46.183 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:37:46.183 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:37:46.184 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:37:46.184 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:37:46.184 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.184 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.184 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:37:46.184 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:37:46.184 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:37:46.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2000: test_pg_schedule_scrub: run_mgr td/ceph-helpers x --mgr_stats_period=1 2026-02-22T22:37:46.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:37:46.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:37:46.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:37:46.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:37:46.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:37:46.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:37:46.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:37:46.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:46.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:46.332 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:46.332 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.333 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.333 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:46.333 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:37:46.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T22:37:46.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2026-02-22T22:37:46.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:37:46.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:37:46.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:37:46.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:37:46.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:37:46.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:37:46.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:37:46.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a32dd5a8-431a-41c4-ad37-9f76f7e69db7 2026-02-22T22:37:46.357 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 a32dd5a8-431a-41c4-ad37-9f76f7e69db7 2026-02-22T22:37:46.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a32dd5a8-431a-41c4-ad37-9f76f7e69db7' 2026-02-22T22:37:46.358 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:37:46.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6hZtpV/6jFRAArd9d2YG+6D4kZ3q644pCDQ== 2026-02-22T22:37:46.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6hZtpV/6jFRAArd9d2YG+6D4kZ3q644pCDQ=="}' 2026-02-22T22:37:46.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a32dd5a8-431a-41c4-ad37-9f76f7e69db7 -i td/ceph-helpers/0/new.json 2026-02-22T22:37:46.468 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:37:46.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:37:46.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6hZtpV/6jFRAArd9d2YG+6D4kZ3q644pCDQ== --osd-uuid a32dd5a8-431a-41c4-ad37-9f76f7e69db7 2026-02-22T22:37:46.494 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:46.492+0000 7f616d7d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:46.495 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:46.493+0000 7f616d7d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:46.496 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:46.494+0000 7f616d7d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:46.496 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:46.495+0000 7f616d7d4780 -1 bdev(0x5561e8aa3c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:37:46.497 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:46.495+0000 7f616d7d4780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:37:49.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:37:49.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:37:49.385 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:37:49.385 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:37:49.385 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:37:49.640 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:37:49.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:37:49.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:37:49.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:37:49.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:37:49.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:37:49.657 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:49.655+0000 7fb425c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:49.658 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:49.657+0000 7fb425c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:49.659 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:49.658+0000 7fb425c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:37:49.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:37:49.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:37:49.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:37:49.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:37:49.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:37:49.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:49.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:37:49.831 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:37:49.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:49.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:50.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:51.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:51.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:51.023 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:37:51.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:37:51.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:51.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:51.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:51.247 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:51.245+0000 7fb425c1c780 -1 Falling back to public interface 2026-02-22T22:37:52.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:52.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:52.211 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:37:52.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:37:52.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:52.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:52.400 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:53.401 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:53.401 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:53.401 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:37:53.402 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:37:53.402 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:53.402 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:53.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:54.193 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:37:54.191+0000 7fb425c1c780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:37:54.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:54.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:54.589 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:37:54.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:37:54.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:54.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:54.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:37:55.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:37:55.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:37:55.778 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:37:55.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:37:55.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:37:55.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:37:55.971 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3013367654,v1:127.0.0.1:6803/3013367654] [v2:127.0.0.1:6804/3013367654,v1:127.0.0.1:6805/3013367654] exists,up a32dd5a8-431a-41c4-ad37-9f76f7e69db7 2026-02-22T22:37:55.972 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:37:55.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:37:55.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:37:55.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: test_pg_schedule_scrub: create_rbd_pool 2026-02-22T22:37:55.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:37:56.158 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:37:56.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:37:56.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:37:56.402 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:37:56.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:37:57.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:37:57.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2003: test_pg_schedule_scrub: wait_for_clean 2026-02-22T22:37:57.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:37:57.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:37:57.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:37:57.695 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:37:57.695 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:37:57.695 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:37:57.695 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:37:57.695 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:37:57.695 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:37:57.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:37:57.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:37:57.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:37:57.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:37:57.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:37:57.776 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:37:57.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:37:57.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:37:57.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:37:57.963 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:37:58.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836485 2026-02-22T22:37:58.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836485 2026-02-22T22:37:58.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T22:37:58.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:37:58.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836485 2026-02-22T22:37:58.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:37:58.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:37:58.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836485 2026-02-22T22:37:58.027 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:37:58.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836485 2026-02-22T22:37:58.028 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T22:37:58.029 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836485 2026-02-22T22:37:58.029 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:37:58.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836485 2026-02-22T22:37:58.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:37:59.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:37:59.228 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:37:59.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-22T22:37:59.417 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:37:59.417 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:37:59.418 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:37:59.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:37:59.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:37:59.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:37:59.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:37:59.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:37:59.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:37:59.668 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:37:59.668 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:37:59.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:37:59.852 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:37:59.852 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:37:59.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:38:00.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:38:00.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:38:00.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:38:00.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2004: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2026-02-22T22:38:00.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: pg_schedule_scrub: local pgid=1.0 2026-02-22T22:38:00.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: pg_schedule_scrub: wait_for_pg_clean 1.0 2026-02-22T22:38:00.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: local pg_id=1.0 2026-02-22T22:38:00.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T22:38:00.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:38:00.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:38:00.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:38:00.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:38:00.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:38:00.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T22:38:00.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: local -a delays 2026-02-22T22:38:00.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1713: wait_for_pg_clean: local -i loop=0 2026-02-22T22:38:00.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_pg_clean: flush_pg_stats 2026-02-22T22:38:00.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:38:00.307 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:38:00.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:38:00.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:38:00.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:38:00.498 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:38:00.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836490 2026-02-22T22:38:00.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836490 2026-02-22T22:38:00.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836490' 2026-02-22T22:38:00.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:38:00.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836490 2026-02-22T22:38:00.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:38:00.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:38:00.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836490 2026-02-22T22:38:00.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:38:00.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836490 2026-02-22T22:38:00.563 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836490 2026-02-22T22:38:00.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-22T22:38:00.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:00.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836487 -lt 21474836490 2026-02-22T22:38:00.753 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:38:01.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:38:01.754 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:01.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836487 -lt 21474836490 2026-02-22T22:38:01.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:38:02.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:38:02.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:03.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-22T22:38:03.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:38:03.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T22:38:03.138 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 0 2026-02-22T22:38:03.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:38:03.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:38:03.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:38:03.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:38:03.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state=active+clean 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: break 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_pg_clean: return 0 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: pg_schedule_scrub: get_last_scrub_stamp 1.0 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:03.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:03.207 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:03.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: pg_schedule_scrub: local last_scrub=2026-02-22T22:37:56.352683+0000 2026-02-22T22:38:03.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: pg_schedule_scrub: ceph tell 1.0 schedule-scrub 2026-02-22T22:38:03.451 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:38:03.451 INFO:tasks.workunit.client.0.trial123.stdout: "deep": false, 2026-02-22T22:38:03.451 INFO:tasks.workunit.client.0.trial123.stdout: "must": false, 2026-02-22T22:38:03.451 INFO:tasks.workunit.client.0.trial123.stdout: "stamp": "2026-02-15T22:36:23.449959+0000" 2026-02-22T22:38:03.451 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: pg_schedule_scrub: wait_for_scrub 1.0 2026-02-22T22:37:56.352683+0000 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: wait_for_scrub: local pgid=1.0 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: local last_scrub=2026-02-22T22:37:56.352683+0000 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i=0 )) 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:03.456 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:03.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:03.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:03.457 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:03.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:37:56.352683+0000 '>' 2026-02-22T22:37:56.352683+0000 2026-02-22T22:38:03.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:38:04.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:38:04.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:04.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:04.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:04.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:04.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:04.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:04.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:37:56.352683+0000 '>' 2026-02-22T22:37:56.352683+0000 2026-02-22T22:38:04.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:38:05.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:38:05.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:05.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:05.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:05.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:05.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:05.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:06.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:37:56.352683+0000 '>' 2026-02-22T22:37:56.352683+0000 2026-02-22T22:38:06.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:38:07.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:38:07.015 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:07.015 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:07.015 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:07.016 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:07.016 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:07.016 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:07.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:38:04.252207+0000 '>' 2026-02-22T22:37:56.352683+0000 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: wait_for_scrub: return 0 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_pg_schedule_scrub: teardown td/ceph-helpers 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:38:07.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:38:07.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:38:07.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:38:07.313 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:38:07.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:38:07.314 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:38:07.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:38:07.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:38:07.315 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:38:07.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:38:07.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:38:07.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:38:07.317 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:38:07.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:38:07.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:38:07.319 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:38:07.322 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:38:07.322 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.322 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:38:07.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:38:07.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:38:07.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:38:07.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_pg_scrub td/ceph-helpers 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_scrub: local dir=td/ceph-helpers 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1952: test_pg_scrub: setup td/ceph-helpers 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:38:07.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:38:07.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:38:07.326 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:38:07.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:38:07.327 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:38:07.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:38:07.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:38:07.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:38:07.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:38:07.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:38:07.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:38:07.329 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:38:07.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:38:07.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:38:07.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:38:07.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:38:07.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:38:07.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:38:07.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:38:07.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:38:07.335 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:38:07.335 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.335 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:38:07.336 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:38:07.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:38:07.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1953: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:38:07.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:38:07.356 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:38:07.356 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:38:07.356 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:38:07.356 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:38:07.356 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.357 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:38:07.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:38:07.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:38:07.385 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:38:07.386 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.386 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.386 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:38:07.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:38:07.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:38:07.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:38:07.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:38:07.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.430 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:38:07.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:38:07.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:38:07.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: test_pg_scrub: run_mgr td/ceph-helpers x 2026-02-22T22:38:07.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:38:07.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:38:07.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:38:07.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:38:07.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:38:07.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:38:07.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:38:07.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:38:07.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:38:07.574 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:38:07.574 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.574 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:38:07.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:38:07.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:38:07.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1955: test_pg_scrub: run_osd td/ceph-helpers 0 2026-02-22T22:38:07.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:38:07.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:38:07.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:38:07.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:38:07.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:38:07.596 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:38:07.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:38:07.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:38:07.598 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:38:07.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=81e079fa-3b4d-4165-b4ce-73d154d1d13e 2026-02-22T22:38:07.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 81e079fa-3b4d-4165-b4ce-73d154d1d13e' 2026-02-22T22:38:07.599 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 81e079fa-3b4d-4165-b4ce-73d154d1d13e 2026-02-22T22:38:07.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:38:07.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPhZtpdacdJBAA/Sa8RDY0MR6Su8HfUujDNw== 2026-02-22T22:38:07.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPhZtpdacdJBAA/Sa8RDY0MR6Su8HfUujDNw=="}' 2026-02-22T22:38:07.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 81e079fa-3b4d-4165-b4ce-73d154d1d13e -i td/ceph-helpers/0/new.json 2026-02-22T22:38:07.712 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:38:07.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:38:07.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPhZtpdacdJBAA/Sa8RDY0MR6Su8HfUujDNw== --osd-uuid 81e079fa-3b4d-4165-b4ce-73d154d1d13e 2026-02-22T22:38:07.738 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:07.735+0000 7f844dec0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:38:07.739 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:07.736+0000 7f844dec0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:38:07.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:07.737+0000 7f844dec0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:38:07.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:07.737+0000 7f844dec0780 -1 bdev(0x55da71787c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:38:07.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:07.737+0000 7f844dec0780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:38:10.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:38:10.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:38:10.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:38:10.630 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:38:10.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:38:10.885 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:38:10.885 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:38:10.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:38:10.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:38:10.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:38:10.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:38:10.903 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:10.900+0000 7ff641993780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:38:10.904 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:10.901+0000 7ff641993780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:38:10.905 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:10.902+0000 7ff641993780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:38:11.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:38:11.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:38:11.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:38:11.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:38:11.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:38:11.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:38:11.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:38:11.075 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:38:11.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:38:11.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:38:11.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:38:12.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:38:12.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:38:12.266 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:38:12.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:38:12.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:38:12.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:38:12.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:38:12.496 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:12.493+0000 7ff641993780 -1 Falling back to public interface 2026-02-22T22:38:13.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:38:13.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:38:13.455 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:38:13.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:38:13.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:38:13.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:38:13.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:38:14.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:38:14.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:38:14.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:38:14.647 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:38:14.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:38:14.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:38:14.836 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:38:15.187 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:38:15.185+0000 7ff641993780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:38:15.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:38:15.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:38:15.838 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:38:15.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:38:15.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:38:15.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:38:16.029 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1538287505,v1:127.0.0.1:6803/1538287505] [v2:127.0.0.1:6804/1538287505,v1:127.0.0.1:6805/1538287505] exists,up 81e079fa-3b4d-4165-b4ce-73d154d1d13e 2026-02-22T22:38:16.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:38:16.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:38:16.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:38:16.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: test_pg_scrub: create_rbd_pool 2026-02-22T22:38:16.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:38:16.212 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:38:16.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:38:16.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:38:16.494 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:38:16.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:38:17.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: test_pg_scrub: wait_for_clean 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:38:17.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:38:17.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:38:17.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:38:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:38:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:38:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:38:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:38:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:38:17.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:38:18.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:38:18.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:38:18.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:38:18.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:38:18.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:38:18.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:38:18.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:38:18.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:38:18.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:38:18.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:38:18.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:38:18.122 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:38:18.122 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:38:18.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:38:18.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:38:18.124 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:38:18.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:18.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:38:18.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:38:19.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:38:19.314 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:19.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:38:19.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:38:20.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:38:20.507 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:20.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:38:20.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:38:20.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:38:20.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:38:20.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:38:20.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:38:20.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:38:20.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:38:20.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:38:20.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:38:20.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:38:20.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:38:21.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:38:21.137 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:38:21.138 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:38:21.138 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1958: test_pg_scrub: pg_scrub 1.0 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_scrub: local pgid=1.0 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_scrub: wait_for_pg_clean 1.0 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: local pg_id=1.0 2026-02-22T22:38:21.394 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T22:38:21.395 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:38:21.395 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:38:21.395 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:38:21.395 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:38:21.395 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:38:21.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T22:38:21.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: local -a delays 2026-02-22T22:38:21.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1713: wait_for_pg_clean: local -i loop=0 2026-02-22T22:38:21.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_pg_clean: flush_pg_stats 2026-02-22T22:38:21.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:38:21.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:38:21.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:38:21.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:38:21.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:38:21.782 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:38:21.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836486 2026-02-22T22:38:21.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836486 2026-02-22T22:38:21.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T22:38:21.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:38:21.846 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836486 2026-02-22T22:38:21.846 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:38:21.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:38:21.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836486 2026-02-22T22:38:21.847 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:38:21.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836486 2026-02-22T22:38:21.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T22:38:21.849 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836486 2026-02-22T22:38:21.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:22.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-22T22:38:22.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:38:23.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:38:23.040 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:23.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-22T22:38:23.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:38:24.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:38:24.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:38:24.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T22:38:24.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:38:24.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T22:38:24.424 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 0 2026-02-22T22:38:24.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:38:24.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:38:24.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:38:24.425 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:38:24.425 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state=active+clean 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: break 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1728: wait_for_pg_clean: return 0 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_scrub: get_last_scrub_stamp 1.0 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:24.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:24.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_scrub: local last_scrub=2026-02-22T22:38:16.443485+0000 2026-02-22T22:38:24.673 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_scrub: ceph pg scrub 1.0 2026-02-22T22:38:24.850 INFO:tasks.workunit.client.0.trial123.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-22T22:38:24.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_scrub: wait_for_scrub 1.0 2026-02-22T22:38:16.443485+0000 2026-02-22T22:38:24.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: wait_for_scrub: local pgid=1.0 2026-02-22T22:38:24.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: local last_scrub=2026-02-22T22:38:16.443485+0000 2026-02-22T22:38:24.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T22:38:24.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i=0 )) 2026-02-22T22:38:24.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:24.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:24.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:24.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:24.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:24.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:25.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:38:16.443485+0000 '>' 2026-02-22T22:38:16.443485+0000 2026-02-22T22:38:25.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:38:26.043 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:38:26.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:26.044 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:26.044 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:26.044 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:26.045 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:26.045 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:26.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:38:16.443485+0000 '>' 2026-02-22T22:38:16.443485+0000 2026-02-22T22:38:26.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:38:27.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:38:27.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:27.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:27.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:27.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:27.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:27.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:27.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:38:16.443485+0000 '>' 2026-02-22T22:38:16.443485+0000 2026-02-22T22:38:27.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:38:28.428 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:38:28.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:38:28.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:38:28.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:38:28.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:38:28.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:38:28.430 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:38:25.152241+0000 '>' 2026-02-22T22:38:16.443485+0000 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: wait_for_scrub: return 0 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1959: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:38:28.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:38:28.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:38:28.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1960: test_pg_scrub: TIMEOUT=1 2026-02-22T22:38:28.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1960: test_pg_scrub: pg_scrub 1.0 2026-02-22T22:38:28.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_scrub: local pgid=1.0 2026-02-22T22:38:28.724 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_scrub: wait_for_pg_clean 1.0 2026-02-22T22:38:28.724 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: local pg_id=1.0 2026-02-22T22:38:28.724 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T22:38:28.725 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:38:28.725 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:38:28.725 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:38:28.725 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:38:28.725 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:38:28.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T22:38:28.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: local -a delays 2026-02-22T22:38:28.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1713: wait_for_pg_clean: local -i loop=0 2026-02-22T22:38:28.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_pg_clean: flush_pg_stats 2026-02-22T22:38:28.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=1 2026-02-22T22:38:28.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:38:29.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:38:29.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:38:29.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:38:29.112 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:53:17.416 INFO:tasks.workunit.client.0.trial123.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-22T22:53:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq= 2026-02-22T22:53:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z '' 2026-02-22T22:53:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: continue 2026-02-22T22:53:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:17.424 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T22:53:17.424 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 0 2026-02-22T22:53:17.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:17.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:17.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:17.425 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:17.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:17.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:17.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:17.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 0 >= 31 )) 2026-02-22T22:53:17.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 1 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 1 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 1' 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:18.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:18.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:18.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:18.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 1 >= 31 )) 2026-02-22T22:53:18.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 2 2026-02-22T22:53:20.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:20.529 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:20.529 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 2 2026-02-22T22:53:20.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 2' 2026-02-22T22:53:20.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:20.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:20.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:20.530 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:20.530 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:20.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:20.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:20.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 2 >= 31 )) 2026-02-22T22:53:20.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:23.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:23.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:23.585 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 3 2026-02-22T22:53:23.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 3' 2026-02-22T22:53:23.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:23.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:23.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:23.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:23.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:23.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:23.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:23.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 3 >= 31 )) 2026-02-22T22:53:23.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:26.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:26.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:26.638 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 4 2026-02-22T22:53:26.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 4' 2026-02-22T22:53:26.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:26.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:26.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:26.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:26.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:26.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:26.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:26.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 4 >= 31 )) 2026-02-22T22:53:26.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:29.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:29.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:29.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 5' 2026-02-22T22:53:29.689 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 5 2026-02-22T22:53:29.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:29.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:29.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:29.690 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:29.690 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:29.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:29.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:29.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 5 >= 31 )) 2026-02-22T22:53:29.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:32.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:32.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:32.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 6' 2026-02-22T22:53:32.746 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 6 2026-02-22T22:53:32.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:32.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:32.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:32.747 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:32.747 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:32.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:32.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:32.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 6 >= 31 )) 2026-02-22T22:53:32.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:35.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:35.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:35.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 7' 2026-02-22T22:53:35.798 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 7 2026-02-22T22:53:35.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:35.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:35.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:35.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:35.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:35.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:35.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:35.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 7 >= 31 )) 2026-02-22T22:53:35.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:38.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:38.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:38.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 8' 2026-02-22T22:53:38.853 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 8 2026-02-22T22:53:38.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:38.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:38.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:38.853 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:38.854 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 8 >= 31 )) 2026-02-22T22:53:38.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:41.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:41.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:41.909 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 9' 2026-02-22T22:53:41.909 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 9 2026-02-22T22:53:41.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:41.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:41.910 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:41.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:41.911 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:41.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:41.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:41.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 9 >= 31 )) 2026-02-22T22:53:41.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:44.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:44.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:44.964 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 10 2026-02-22T22:53:44.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 10' 2026-02-22T22:53:44.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:44.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:44.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:44.966 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:44.966 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:45.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:45.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:45.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 10 >= 31 )) 2026-02-22T22:53:45.018 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:48.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:48.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:48.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 11' 2026-02-22T22:53:48.019 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 11 2026-02-22T22:53:48.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:48.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:48.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:48.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:48.021 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:48.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:48.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:48.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 11 >= 31 )) 2026-02-22T22:53:48.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:51.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:51.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:51.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 12' 2026-02-22T22:53:51.072 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 12 2026-02-22T22:53:51.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:51.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:51.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:51.073 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:51.073 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:51.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:51.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:51.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 12 >= 31 )) 2026-02-22T22:53:51.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:54.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:54.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:54.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 13' 2026-02-22T22:53:54.125 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 13 2026-02-22T22:53:54.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:54.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:54.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:54.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:54.126 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:54.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:54.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:54.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 13 >= 31 )) 2026-02-22T22:53:54.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:53:57.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:53:57.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:53:57.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 14' 2026-02-22T22:53:57.180 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 14 2026-02-22T22:53:57.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:53:57.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:53:57.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:53:57.181 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:53:57.181 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:53:57.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:53:57.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:53:57.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 14 >= 31 )) 2026-02-22T22:53:57.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:00.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:00.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:00.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 15' 2026-02-22T22:54:00.236 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 15 2026-02-22T22:54:00.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:00.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:00.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:00.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:00.237 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:00.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:00.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:00.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 15 >= 31 )) 2026-02-22T22:54:00.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:03.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:03.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:03.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 16' 2026-02-22T22:54:03.291 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 16 2026-02-22T22:54:03.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:03.291 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:03.292 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:03.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:03.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:03.342 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:03.342 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:03.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 16 >= 31 )) 2026-02-22T22:54:03.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:06.342 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:06.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:06.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 17' 2026-02-22T22:54:06.343 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 17 2026-02-22T22:54:06.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:06.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:06.343 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:06.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:06.344 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:06.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:06.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:06.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 17 >= 31 )) 2026-02-22T22:54:06.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:09.398 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:09.398 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:09.398 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 18' 2026-02-22T22:54:09.399 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 18 2026-02-22T22:54:09.399 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:09.400 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:09.400 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:09.400 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:09.400 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:09.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:09.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:09.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 18 >= 31 )) 2026-02-22T22:54:09.451 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:12.451 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:12.451 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:12.451 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 19' 2026-02-22T22:54:12.451 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 19 2026-02-22T22:54:12.452 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:12.452 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:12.452 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:12.452 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:12.452 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:12.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:12.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:12.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 19 >= 31 )) 2026-02-22T22:54:12.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:15.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:15.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:15.506 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 20 2026-02-22T22:54:15.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 20' 2026-02-22T22:54:15.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:15.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:15.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:15.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:15.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:15.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:15.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:15.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 20 >= 31 )) 2026-02-22T22:54:15.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:18.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:18.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:18.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 21' 2026-02-22T22:54:18.558 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 21 2026-02-22T22:54:18.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:18.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:18.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:18.559 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:18.559 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 21 >= 31 )) 2026-02-22T22:54:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:21.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:21.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:21.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 22' 2026-02-22T22:54:21.613 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 22 2026-02-22T22:54:21.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:21.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:21.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:21.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:21.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 22 >= 31 )) 2026-02-22T22:54:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:24.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:24.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:24.666 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 23 2026-02-22T22:54:24.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 23' 2026-02-22T22:54:24.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:24.668 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:24.668 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:24.668 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:24.668 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:24.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:24.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:24.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 23 >= 31 )) 2026-02-22T22:54:24.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:27.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:27.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:27.721 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 24' 2026-02-22T22:54:27.722 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 24 2026-02-22T22:54:27.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:27.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:27.723 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:27.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:27.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:27.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:27.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:27.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 24 >= 31 )) 2026-02-22T22:54:27.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:30.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:30.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:30.777 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 25' 2026-02-22T22:54:30.777 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 25 2026-02-22T22:54:30.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:30.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:30.778 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:30.778 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:30.778 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:30.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:30.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:30.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 25 >= 31 )) 2026-02-22T22:54:30.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:33.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:33.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:33.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 26' 2026-02-22T22:54:33.832 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 26 2026-02-22T22:54:33.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:33.832 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:33.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:33.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:33.833 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:33.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 26 >= 31 )) 2026-02-22T22:54:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 27' 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 27 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:36.884 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:36.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:36.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:36.937 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 27 >= 31 )) 2026-02-22T22:54:36.938 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:39.938 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:39.938 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:39.938 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 28' 2026-02-22T22:54:39.939 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 28 2026-02-22T22:54:39.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:39.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:39.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:39.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:39.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:39.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:39.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:39.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 28 >= 31 )) 2026-02-22T22:54:39.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:42.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:42.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:42.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 29' 2026-02-22T22:54:42.993 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 29 2026-02-22T22:54:42.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:42.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:42.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:42.994 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:42.994 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:43.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:43.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:43.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 29 >= 31 )) 2026-02-22T22:54:43.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:46.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:46.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:46.046 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 30 2026-02-22T22:54:46.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 30' 2026-02-22T22:54:46.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:46.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:46.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:46.047 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:46.047 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:46.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:46.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:46.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 30 >= 31 )) 2026-02-22T22:54:46.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1725: wait_for_pg_clean: sleep 3 2026-02-22T22:54:49.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1726: wait_for_pg_clean: loop+=1 2026-02-22T22:54:49.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: true 2026-02-22T22:54:49.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1718: wait_for_pg_clean: echo '#---------- 1.0 loop 31' 2026-02-22T22:54:49.101 INFO:tasks.workunit.client.0.trial123.stdout:#---------- 1.0 loop 31 2026-02-22T22:54:49.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T22:54:49.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: is_pg_clean: local pgid=1.0 2026-02-22T22:54:49.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: is_pg_clean: local pg_state 2026-02-22T22:54:49.101 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: ceph pg 1.0 query 2026-02-22T22:54:49.101 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: jq -r '.state ' 2026-02-22T22:54:49.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: is_pg_clean: pg_state= 2026-02-22T22:54:49.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T22:54:49.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1720: wait_for_pg_clean: (( 31 >= 31 )) 2026-02-22T22:54:49.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1721: wait_for_pg_clean: ceph report 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_fingerprint": "6f559326-50b7-48ad-b635-d375154dbfd0", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "version": "19.2.3-689-gf90f4eae", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "commit": "f90f4eaebb28b619cf76e80cbbe997656da69579", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "timestamp": "2026-02-22T22:54:49.399607+0000", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "tag": "", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "health": { 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "status": "HEALTH_WARN", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "checks": { 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "OSD_DOWN": { 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "message": "1 osds down", 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "count": 1 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [ 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.401 INFO:tasks.workunit.client.0.trial123.stdout: "message": "osd.0 (root=default,host=trial123) is down" 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "OSD_HOST_DOWN": { 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "message": "1 host (1 osds) down", 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "count": 1 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [ 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "message": "host trial123 (root=default) (1 osds) is down" 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "OSD_ROOT_DOWN": { 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "message": "1 root (1 osds) down", 2026-02-22T22:54:49.402 INFO:tasks.workunit.client.0.trial123.stdout: "count": 1 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [ 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "message": "root default (1 osds) is down" 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "PG_AVAILABILITY": { 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "message": "Reduced data availability: 4 pgs stale", 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "count": 4 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [ 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.0 is stuck stale for 90s, current state stale+active+clean, last acting [0]" 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.1 is stuck stale for 90s, current state stale+active+clean, last acting [0]" 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.403 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.2 is stuck stale for 90s, current state stale+active+clean, last acting [0]" 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.3 is stuck stale for 90s, current state stale+active+clean, last acting [0]" 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "POOL_NO_REDUNDANCY": { 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "message": "1 pool(s) have no replicas configured", 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "count": 1 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [ 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.404 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pool 'rbd' has no replicas configured" 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "TOO_FEW_OSDS": { 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "count": 2 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [], 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.405 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "mutes": [] 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "monmap_first_committed": 1, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "monmap_last_committed": 1, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "monmap": { 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "fsid": "d116af49-5d19-4d5a-91be-c8a20698ede8", 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:38:07.350063Z", 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "created": "2026-02-22T22:38:07.350063Z", 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "min_mon_release": 19, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "min_mon_release_name": "squid", 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "election_strategy": 1, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "disallowed_leaders": "", 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode": false, 2026-02-22T22:54:49.406 INFO:tasks.workunit.client.0.trial123.stdout: "tiebreaker_mon": "", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "removed_ranks": "", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "features": { 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "persistent": [ 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "kraken", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "luminous", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "mimic", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap-prune", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "nautilus", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "octopus", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "pacific", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "elector-pinging", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "quincy", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "reef", 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: "squid" 2026-02-22T22:54:49.407 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "optional": [] 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "mons": [ 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "rank": 0, 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "name": "a", 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "public_addrs": { 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:7109", 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 0 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.408 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:7109/0", 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "public_addr": "127.0.0.1:7109/0", 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "priority": 0, 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 0, 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "crush_location": "{}" 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "quorum": [ 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: 0 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap": { 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 14, 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "fsid": "d116af49-5d19-4d5a-91be-c8a20698ede8", 2026-02-22T22:54:49.409 INFO:tasks.workunit.client.0.trial123.stdout: "created": "2026-02-22T22:38:07.384281+0000", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:53:17.413453+0000", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "last_up_change": "2026-02-22T22:53:17.413453+0000", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "last_in_change": "2026-02-22T22:38:07.688880+0000", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "flags_num": 5799936, 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "flags_set": [ 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "pglog_hardlimit", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snapdirs", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "recovery_deletes", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "sortbitwise" 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "crush_version": 3, 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "full_ratio": 0.99000000953674316, 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "backfillfull_ratio": 0.99000000953674316, 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "nearfull_ratio": 0.85000002384185791, 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_snapshot": "", 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "pool_max": 1, 2026-02-22T22:54:49.410 INFO:tasks.workunit.client.0.trial123.stdout: "max_osd": 1, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "require_min_compat_client": "luminous", 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "min_compat_client": "jewel", 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "require_osd_release": "squid", 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "allow_crimson": false, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pools": [ 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pool_name": "rbd", 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "create_time": "2026-02-22T22:38:16.359591+0000", 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "flags": 8193, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "type": 1, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "size": 1, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "min_size": 1, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "crush_rule": 0, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_count": 0, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_target": 0, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_barrier": 0, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "is_stretch_pool": false, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "object_hash": 2, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pg_autoscale_mode": "off", 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num": 4, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pg_placement_num": 4, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pg_placement_num_target": 4, 2026-02-22T22:54:49.411 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num_target": 4, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num_pending": 4, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "last_pg_merge_meta": { 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "source_pgid": "0.0", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "ready_epoch": 0, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_started": 0, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_clean": 0, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "source_version": "0'0", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "target_version": "0'0" 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "last_change": "12", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend": "0", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend_prenautilus": "0", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend_preluminous": "0", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "auid": 0, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "snap_mode": "selfmanaged", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "snap_seq": 2, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "snap_epoch": 12, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "pool_snaps": [], 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "removed_snaps": "[]", 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "quota_max_bytes": 0, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "quota_max_objects": 0, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "tiers": [], 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "tier_of": -1, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "read_tier": -1, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "write_tier": -1, 2026-02-22T22:54:49.412 INFO:tasks.workunit.client.0.trial123.stdout: "cache_mode": "none", 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "target_max_bytes": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "target_max_objects": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_dirty_ratio_micro": 400000, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_full_ratio_micro": 800000, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "cache_min_flush_age": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "cache_min_evict_age": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "erasure_code_profile": "", 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_params": { 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "type": "none" 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_period": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_count": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "use_gmt_hitset": true, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "min_read_recency_for_promote": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "min_write_recency_for_promote": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_grade_decay_rate": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_search_last_n": 0, 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "grade_table": [], 2026-02-22T22:54:49.413 INFO:tasks.workunit.client.0.trial123.stdout: "stripe_width": 0, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "expected_num_objects": 0, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "fast_read": false, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "options": {}, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "application_metadata": { 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "rbd": {} 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "osds": [ 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "uuid": "81e079fa-3b4d-4165-b4ce-73d154d1d13e", 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "up": 0, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "in": 1, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 1, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "primary_affinity": 1, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "last_clean_begin": 0, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "last_clean_end": 0, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 5, 2026-02-22T22:54:49.414 INFO:tasks.workunit.client.0.trial123.stdout: "up_thru": 7, 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "down_at": 14, 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "lost_at": 0, 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "public_addrs": { 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6802", 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6803", 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_addrs": { 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:54:49.415 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6804", 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6805", 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_back_addrs": { 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6808", 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6809", 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.416 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_front_addrs": { 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6806", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6807", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 1538287505 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "public_addr": "127.0.0.1:6803/1538287505", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_addr": "127.0.0.1:6805/1538287505", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1538287505", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1538287505", 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "state": [ 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: "exists" 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.417 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "osd_xinfo": [ 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "down_stamp": "2026-02-22T22:53:17.413453+0000", 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "laggy_probability": 0, 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "laggy_interval": 0, 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "features": 4540701547738038271, 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "old_weight": 0, 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "last_purged_snaps_scrub": "2026-02-22T22:38:16.139136+0000", 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "dead_epoch": 0 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap": [], 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap_items": [], 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap_primaries": [], 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "pg_temp": [], 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "primary_temp": [], 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "blocklist": {}, 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "range_blocklist": {}, 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "erasure_code_profiles": { 2026-02-22T22:54:49.418 INFO:tasks.workunit.client.0.trial123.stdout: "default": { 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "k": "2", 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "m": "2", 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "plugin": "jerasure", 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "technique": "reed_sol_van" 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "removed_snaps_queue": [], 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "new_removed_snaps": [], 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "new_purged_snaps": [], 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "crush_node_flags": {}, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "device_class_flags": {}, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode": { 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode_enabled": false, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_bucket_count": 0, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "degraded_stretch_mode": 0, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "recovering_stretch_mode": 0, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode_bucket": 0 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: "osd_metadata": [ 2026-02-22T22:54:49.419 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "arch": "x86_64", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "back_addr": "[v2:127.0.0.1:6804/1538287505,v1:127.0.0.1:6805/1538287505]", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "back_iface": "", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs": "1", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_dedicated_db": "0", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_dedicated_wal": "0", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_single_shared_device": "1", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_allocation_from_file": "1", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_allocator": "hybrid", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_access_mode": "file", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_block_size": "4096", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_devices": "nvme1n1", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_driver": "KernelDevice", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_optimal_io_size": "4096", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_rotational": "0", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_size": "107374182400", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_support_discard": "1", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_type": "ssd", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_min_alloc_size": "4096", 2026-02-22T22:54:49.420 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_release": "squid", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version": "ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable)", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version_short": "19.2.3-689-gf90f4eae", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version_when_created": "ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable)", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "cpu": "AMD EPYC 4464P 12-Core Processor", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "created_at": "2026-02-22T22:38:10.109022Z", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "default_device_class": "ssd", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "device_ids": "nvme1n1=Micron_7450_MTFDKBA960TFR_25174FCD7341", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "device_paths": "nvme1n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "devices": "nvme1n1", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "distro": "centos", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "distro_description": "CentOS Stream 9", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "distro_version": "9", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "front_addr": "[v2:127.0.0.1:6802/1538287505,v1:127.0.0.1:6803/1538287505]", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "front_iface": "", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1538287505,v1:127.0.0.1:6809/1538287505]", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1538287505,v1:127.0.0.1:6807/1538287505]", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "hostname": "trial123", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "journal_rotational": "0", 2026-02-22T22:54:49.421 INFO:tasks.workunit.client.0.trial123.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Feb 11 20:19:22 UTC 2026", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "kernel_version": "5.14.0-681.el9.x86_64", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "mem_swap_kb": "8388604", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "mem_total_kb": "130438940", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "objectstore_numa_node": "0", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "objectstore_numa_nodes": "0", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "os": "Linux", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "osd_data": "td/ceph-helpers/0", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "osd_objectstore": "bluestore", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "osdspec_affinity": "", 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "rotational": "0" 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_clean_epochs": { 2026-02-22T22:54:49.422 INFO:tasks.workunit.client.0.trial123.stdout: "min_last_epoch_clean": 0, 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_clean": { 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "per_pool": [] 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "osd_epochs": [] 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_first_committed": 1, 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_last_committed": 14, 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "crushmap": { 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "devices": [ 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "name": "osd.0", 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: "class": "ssd" 2026-02-22T22:54:49.423 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: "types": [ 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 0, 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: "name": "osd" 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: "name": "host" 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 2, 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: "name": "chassis" 2026-02-22T22:54:49.424 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 3, 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: "name": "rack" 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 4, 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: "name": "row" 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 5, 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: "name": "pdu" 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.425 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 6, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "name": "pod" 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 7, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "name": "room" 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 8, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "name": "datacenter" 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 9, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: "name": "zone" 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.426 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 10, 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "name": "region" 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "name": "root" 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "buckets": [ 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "id": -1, 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "name": "default", 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "root", 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:54:49.427 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "id": -3, 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "id": -2, 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "name": "default~ssd", 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "root", 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:54:49.428 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "id": -4, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "id": -3, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "name": "trial123", 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "host", 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.429 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "id": -4, 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "name": "trial123~ssd", 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "host", 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "rules": [ 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.430 INFO:tasks.workunit.client.0.trial123.stdout: "rule_id": 0, 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "rule_name": "replicated_rule", 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "type": 1, 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "steps": [ 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "op": "take", 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "item": -1, 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "item_name": "default" 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "op": "choose_firstn", 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "num": 0, 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "type": "osd" 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "op": "emit" 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "tunables": { 2026-02-22T22:54:49.431 INFO:tasks.workunit.client.0.trial123.stdout: "choose_local_tries": 0, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "choose_local_fallback_tries": 0, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "choose_total_tries": 50, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_descend_once": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_vary_r": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_stable": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "msr_descents": 100, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "msr_collision_tries": 100, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "straw_calc_version": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "allowed_bucket_algs": 54, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "profile": "jewel", 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "optimal_tunables": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "legacy_tunables": 0, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "minimum_required_version": "jewel", 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables2": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "has_v2_rules": 0, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables3": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "has_v3_rules": 0, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "has_v4_buckets": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables5": 1, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "has_v5_rules": 0, 2026-02-22T22:54:49.432 INFO:tasks.workunit.client.0.trial123.stdout: "has_msr_rules": 0 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "choose_args": {} 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "fsmap": { 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "btime": "2026-02-22T22:38:07:384102+0000", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "default_fscid": -1, 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "compat": { 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "compat": {}, 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "ro_compat": {}, 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "incompat": { 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_1": "base v0.20", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_2": "client writeable ranges", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_3": "default file layouts on dirs", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_4": "dir inode in separate object", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_5": "mds uses versioned encoding", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_6": "dirfrag is stored in omap", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_8": "no anchor table", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_9": "file layout v2", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_10": "snaprealm v2", 2026-02-22T22:54:49.433 INFO:tasks.workunit.client.0.trial123.stdout: "feature_11": "minor log segments", 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "feature_12": "quiesce subvolumes" 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "feature_flags": { 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "enable_multiple": true, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "ever_enabled_multiple": true 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "standbys": [], 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "filesystems": [] 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "mdsmap_first_committed": 1, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "mdsmap_last_committed": 1, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "auth": { 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "first_committed": 1, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "last_committed": 3, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "num_secrets": 1 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg": 4, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_active": 4, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_unknown": 0, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "num_osd": 1, 2026-02-22T22:54:49.434 INFO:tasks.workunit.client.0.trial123.stdout: "pool_sum": { 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "stat_sum": { 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes": 19, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects": 2, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_clones": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_copies": 2, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_degraded": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_misplaced": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_unfound": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_dirty": 2, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_whiteouts": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_read": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_read_kb": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_write": 2, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_write_kb": 2, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_scrub_errors": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_recovered": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_recovered": 0, 2026-02-22T22:54:49.435 INFO:tasks.workunit.client.0.trial123.stdout: "num_keys_recovered": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_omap": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_kb": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_kb": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_promote": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_high": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_low": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_some": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_full": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_pinned": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_large_omap_objects": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_manifest": 0, 2026-02-22T22:54:49.436 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_bytes": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_keys": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_repaired": 0 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "store_stats": { 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "total": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "available": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:54:49.437 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 0, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 0 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "log_size": 3, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "ondisk_log_size": 3, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "up": 4, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "acting": 4, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "num_store_stats": 0 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "osd_sum": { 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 0, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 0, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "num_pgs": 4, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "num_osds": 1, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_osds": 1, 2026-02-22T22:54:49.438 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "kb": 104857600, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used": 27080, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_data": 116, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_omap": 1, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_meta": 26942, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "kb_avail": 104830520, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "statfs": { 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "total": 107374182400, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "available": 107346452480, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 118784, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 24831, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:54:49.439 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 1594, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 27589062 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "hb_peers": [], 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "snap_trim_queue_len": 0, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "num_snap_trimming": 0, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "num_shards_repaired": 0, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "op_queue_age_hist": { 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "histogram": [], 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "upper_bound": 1 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "perf_stat": { 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ms": 0, 2026-02-22T22:54:49.440 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ms": 0, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ns": 0, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ns": 0 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "alerts": [], 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "network_ping_times": [] 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "osd_sum_by_class": { 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "ssd": { 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 0, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 0, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "num_pgs": 4, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "num_osds": 1, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_osds": 1, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "kb": 104857600, 2026-02-22T22:54:49.441 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used": 27080, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_data": 116, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_omap": 1, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_meta": 26942, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "kb_avail": 104830520, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "statfs": { 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "total": 107374182400, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "available": 107346452480, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 118784, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 24831, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 1594, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 27589062 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.442 INFO:tasks.workunit.client.0.trial123.stdout: "hb_peers": [], 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "snap_trim_queue_len": 0, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "num_snap_trimming": 0, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "num_shards_repaired": 0, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "op_queue_age_hist": { 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "histogram": [], 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "upper_bound": 1 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "perf_stat": { 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ms": 0, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ms": 0, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ns": 0, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ns": 0 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "alerts": [], 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "network_ping_times": [] 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "pool_stats": [ 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "poolid": 1, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg": 4, 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "stat_sum": { 2026-02-22T22:54:49.443 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes": 19, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects": 2, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_clones": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_copies": 2, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_degraded": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_misplaced": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_unfound": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_dirty": 2, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_whiteouts": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_read": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_read_kb": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_write": 2, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_write_kb": 2, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_scrub_errors": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_recovered": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_recovered": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_keys_recovered": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_omap": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_kb": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_kb": 0, 2026-02-22T22:54:49.444 INFO:tasks.workunit.client.0.trial123.stdout: "num_promote": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_high": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_low": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_some": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_full": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_pinned": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_large_omap_objects": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_manifest": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_bytes": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_keys": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_repaired": 0 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "store_stats": { 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "total": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "available": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 4096, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 19, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 0, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 0 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "log_size": 3, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "ondisk_log_size": 3, 2026-02-22T22:54:49.445 INFO:tasks.workunit.client.0.trial123.stdout: "up": 4, 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "acting": 4, 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "num_store_stats": 1 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "osd_stats": [ 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 21474836487 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_by_state": [ 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "state": "stale+active+clean", 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "num": 4 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_by_osd": [ 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:54:49.446 INFO:tasks.workunit.client.0.trial123.stdout: "num_primary_pg": 4, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "num_acting_pg": 4, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "num_up_not_acting_pg": 0 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snaps": [ 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snaps": {} 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "servicemap": { 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 3, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:39:17.569758+0000", 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "services": {} 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "mgrstat_first_committed": 252, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "mgrstat_last_committed": 498, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "logm_first_committed": 1, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "logm_last_committed": 551, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "paxos": { 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "first_committed": 503, 2026-02-22T22:54:49.447 INFO:tasks.workunit.client.0.trial123.stdout: "last_committed": 1075, 2026-02-22T22:54:49.448 INFO:tasks.workunit.client.0.trial123.stdout: "last_pn": 0, 2026-02-22T22:54:49.448 INFO:tasks.workunit.client.0.trial123.stdout: "accepted_pn": 0 2026-02-22T22:54:49.448 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:54:49.448 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:54:49.448 INFO:tasks.workunit.client.0.trial123.stderr:report 2369886356 2026-02-22T22:54:49.448 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1722: wait_for_pg_clean: echo 'PG 1.0 is not clean after 31 iterations' 2026-02-22T22:54:49.448 INFO:tasks.workunit.client.0.trial123.stdout:PG 1.0 is not clean after 31 iterations 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1723: wait_for_pg_clean: return 1 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_scrub: return 1 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1961: test_pg_scrub: teardown td/ceph-helpers 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:54:49.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:54:49.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:54:49.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:54:49.514 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:54:49.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:54:49.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:54:49.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:54:49.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:54:49.517 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:54:49.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:54:49.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:54:49.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:54:49.519 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:54:49.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:54:49.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:54:49.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:54:49.530 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:54:49.531 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.531 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.531 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:54:49.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_repair td/ceph-helpers 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: test_repair: local dir=td/ceph-helpers 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_repair: setup td/ceph-helpers 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:54:49.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:54:49.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:54:49.534 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:54:49.535 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:54:49.535 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:54:49.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:54:49.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:54:49.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:54:49.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:54:49.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:54:49.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:54:49.537 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:54:49.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:54:49.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:54:49.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:54:49.540 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:54:49.540 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.540 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:54:49.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:54:49.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:54:49.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:54:49.542 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:54:49.543 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.543 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:54:49.544 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:54:49.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:54:49.564 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:54:49.564 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:54:49.564 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:54:49.564 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:54:49.565 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.565 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:54:49.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:54:49.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:54:49.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:54:49.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:54:49.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:54:49.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:54:49.594 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:54:49.594 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:54:49.594 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:54:49.594 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:54:49.594 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:54:49.595 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.595 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.595 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:54:49.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:54:49.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:54:49.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:54:49.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:54:49.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:54:49.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:54:49.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:54:49.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:54:49.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:54:49.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:54:49.639 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:54:49.639 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.639 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.640 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:54:49.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:54:49.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:54:49.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: test_repair: run_mgr td/ceph-helpers x 2026-02-22T22:54:49.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:54:49.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:54:49.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:54:49.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:54:49.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:54:49.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:54:49.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:54:49.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:54:49.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:54:49.786 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:54:49.786 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.786 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.787 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:54:49.787 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:54:49.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:54:49.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: test_repair: run_osd td/ceph-helpers 0 2026-02-22T22:54:49.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:54:49.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:54:49.808 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:54:49.809 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:54:49.810 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:54:49.810 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0839d711-0009-4302-ba8f-1b181871b9d2 2026-02-22T22:54:49.810 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0839d711-0009-4302-ba8f-1b181871b9d2' 2026-02-22T22:54:49.810 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 0839d711-0009-4302-ba8f-1b181871b9d2 2026-02-22T22:54:49.810 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:54:49.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5iZtpJnnPMBAA6Zg4aeGSrSxzxAg6gdflBQ== 2026-02-22T22:54:49.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5iZtpJnnPMBAA6Zg4aeGSrSxzxAg6gdflBQ=="}' 2026-02-22T22:54:49.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0839d711-0009-4302-ba8f-1b181871b9d2 -i td/ceph-helpers/0/new.json 2026-02-22T22:54:49.922 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:54:49.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:54:49.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5iZtpJnnPMBAA6Zg4aeGSrSxzxAg6gdflBQ== --osd-uuid 0839d711-0009-4302-ba8f-1b181871b9d2 2026-02-22T22:54:49.946 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:49.945+0000 7fa5dab53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:54:49.948 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:49.946+0000 7fa5dab53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:54:49.948 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:49.947+0000 7fa5dab53780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:54:49.949 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:49.947+0000 7fa5dab53780 -1 bdev(0x56152f975c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:54:49.949 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:49.947+0000 7fa5dab53780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:54:52.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:54:52.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:54:52.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:54:52.353 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:54:52.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:54:52.611 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:54:52.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:54:52.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:54:52.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:54:52.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:54:52.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:54:52.627 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:52.625+0000 7f25caa11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:54:52.628 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:52.627+0000 7f25caa11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:54:52.629 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:52.627+0000 7f25caa11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:54:52.797 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:54:52.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:54:52.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:54:52.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:54:53.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:54:53.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:54:53.989 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:54:53.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:54:53.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:54:53.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:54:54.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:54:54.208 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:54.207+0000 7f25caa11780 -1 Falling back to public interface 2026-02-22T22:54:55.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:54:55.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:54:55.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:54:55.180 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:54:55.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:54:55.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:54:55.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:54:56.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:54:56.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:54:56.370 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:54:56.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:54:56.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:54:56.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:54:56.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:54:56.913 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:54:56.911+0000 7f25caa11780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:54:57.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:54:57.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:54:57.564 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:54:57.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:54:57.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:54:57.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:54:57.756 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3240938126,v1:127.0.0.1:6803/3240938126] [v2:127.0.0.1:6804/3240938126,v1:127.0.0.1:6805/3240938126] exists,up 0839d711-0009-4302-ba8f-1b181871b9d2 2026-02-22T22:54:57.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:54:57.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:54:57.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:54:57.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1909: test_repair: create_rbd_pool 2026-02-22T22:54:57.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:54:57.937 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:54:57.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:54:57.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:54:58.219 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:54:58.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:54:59.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:54:59.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: test_repair: wait_for_clean 2026-02-22T22:54:59.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:54:59.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:54:59.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:54:59.509 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:54:59.510 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:54:59.510 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:54:59.510 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:54:59.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:54:59.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:54:59.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:54:59.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:54:59.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:54:59.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:54:59.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:54:59.591 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:54:59.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:54:59.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:54:59.779 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:54:59.779 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:54:59.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836483 2026-02-22T22:54:59.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836483 2026-02-22T22:54:59.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T22:54:59.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:54:59.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836483 2026-02-22T22:54:59.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:54:59.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:54:59.844 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836483 2026-02-22T22:54:59.844 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:54:59.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836483 2026-02-22T22:54:59.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T22:54:59.845 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836483 2026-02-22T22:54:59.846 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:55:00.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:55:00.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:55:01.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:55:01.037 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:55:01.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-22T22:55:01.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:55:02.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:55:02.227 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:55:02.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T22:55:02.419 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:55:02.420 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:55:02.420 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:55:02.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:55:02.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:55:02.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:55:02.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:55:02.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:55:02.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:55:02.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:55:02.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:55:02.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:55:02.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:55:02.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:55:02.857 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:55:03.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1911: test_repair: repair 1.0 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: repair: local pgid=1.0 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: repair: get_last_scrub_stamp 1.0 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:03.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:03.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: repair: local last_scrub=2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:03.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: repair: ceph pg repair 1.0 2026-02-22T22:55:03.480 INFO:tasks.workunit.client.0.trial123.stderr:instructing pg 1.0 on osd.0 to repair 2026-02-22T22:55:03.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: repair: wait_for_scrub 1.0 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:03.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: wait_for_scrub: local pgid=1.0 2026-02-22T22:55:03.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: local last_scrub=2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:03.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T22:55:03.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i=0 )) 2026-02-22T22:55:03.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:55:03.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:03.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:03.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:03.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:03.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:03.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:54:58.168459+0000 '>' 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:03.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:55:04.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:55:04.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:55:04.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:04.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:04.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:04.678 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:04.678 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:04.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:54:58.168459+0000 '>' 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:04.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:55:05.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:55:05.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:55:05.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:05.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:05.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:05.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:05.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:06.054 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:54:58.168459+0000 '>' 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:06.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:55:07.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:55:07.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:55:07.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:07.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:07.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:07.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:07.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:07.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:54:58.168459+0000 '>' 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:07.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:55:08.246 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:55:08.246 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:55:08.247 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:08.247 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:08.247 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:08.247 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:08.247 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:08.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:54:58.168459+0000 '>' 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:08.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:55:09.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:55:09.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:55:09.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:09.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:09.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:09.433 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:09.434 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:09.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:54:58.168459+0000 '>' 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:09.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:55:10.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:55:10.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:55:10.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:10.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:10.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:10.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:10.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:55:03.881502+0000 '>' 2026-02-22T22:54:58.168459+0000 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: wait_for_scrub: return 0 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1912: test_repair: kill_daemons td/ceph-helpers KILL osd 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:10.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:10.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:10.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1913: test_repair: TIMEOUT=1 2026-02-22T22:55:10.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1913: test_repair: repair 1.0 2026-02-22T22:55:10.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: repair: local pgid=1.0 2026-02-22T22:55:10.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: repair: get_last_scrub_stamp 1.0 2026-02-22T22:55:10.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:10.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:10.920 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:10.920 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:11.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: repair: local last_scrub=2026-02-22T22:55:03.881502+0000 2026-02-22T22:55:11.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: repair: ceph pg repair 1.0 2026-02-22T22:55:11.290 INFO:tasks.workunit.client.0.trial123.stderr:instructing pg 1.0 on osd.0 to repair 2026-02-22T22:55:11.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: repair: wait_for_scrub 1.0 2026-02-22T22:55:03.881502+0000 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: wait_for_scrub: local pgid=1.0 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: local last_scrub=2026-02-22T22:55:03.881502+0000 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i=0 )) 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 1 )) 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:55:11.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:55:11.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:55:03.881502+0000 '>' 2026-02-22T22:55:03.881502+0000 2026-02-22T22:55:11.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 1 )) 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: wait_for_scrub: return 1 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1914: test_repair: teardown td/ceph-helpers 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:55:12.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:12.487 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:12.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:12.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:12.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:12.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:12.594 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:55:12.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:55:12.595 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:55:12.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:55:12.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:55:12.597 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:55:12.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:12.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:55:12.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:55:12.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:12.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:55:12.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:55:12.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:55:12.604 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:55:12.604 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:12.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:12.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:55:12.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:55:12.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:55:12.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:55:12.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_run_in_background td/ceph-helpers 2026-02-22T22:55:12.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: test_run_in_background: local pids 2026-02-22T22:55:12.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2206: test_run_in_background: run_in_background pids sleep 1 2026-02-22T22:55:12.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: run_in_background: local pid_variable=pids 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: run_in_background: shift 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: eval 'pids+=" 141787"' 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: pids+=' 141787' 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: test_run_in_background: run_in_background pids sleep 1 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: run_in_background: local pid_variable=pids 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: run_in_background: shift 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: eval 'pids+=" 141788"' 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: pids+=' 141788' 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sleep 1 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sed 's/^/141790: /' 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sleep 1 2026-02-22T22:55:12.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sed 's/^/141793: /' 2026-02-22T22:55:12.609 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: test_run_in_background: echo 141787 141788 2026-02-22T22:55:12.609 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: test_run_in_background: wc -w 2026-02-22T22:55:12.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: test_run_in_background: test 2 = 2 2026-02-22T22:55:12.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: test_run_in_background: wait 141787 141788 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: return 0 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: return 0 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_run_mon td/ceph-helpers 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:55:13.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:55:13.608 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:13.608 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:13.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:13.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:13.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:13.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:13.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:55:13.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:55:13.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:55:13.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:55:13.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:55:13.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:55:13.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:13.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:55:13.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:55:13.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:13.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:55:13.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:55:13.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:55:13.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:55:13.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:13.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:13.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:55:13.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:55:13.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:55:13.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:55:13.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:55:13.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:13.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:13.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:55:13.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:55:13.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:55:13.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:55:13.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:13.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:13.643 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:13.643 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:13.643 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:13.644 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:13.644 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:55:13.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:55:13.672 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:55:13.672 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:13.672 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:13.672 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:55:13.673 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:55:13.673 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:13.673 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:13.673 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:13.673 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:13.673 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:13.674 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:13.674 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:13.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:13.674 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:55:13.716 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:55:13.716 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:13.716 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:13.716 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:13.717 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:13.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:13.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:55:13.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2026-02-22T22:55:13.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2026-02-22T22:55:13.857 INFO:tasks.workunit.client.0.trial123.stderr:dumped monmap epoch 1 2026-02-22T22:55:13.863 INFO:tasks.workunit.client.0.trial123.stdout:0: v2:127.0.0.1:7109/0 mon.a 2026-02-22T22:55:13.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2026-02-22T22:55:13.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:13.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:13.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:13.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:13.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:14.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:14.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2026-02-22T22:55:14.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:55:14.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:55:14.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:55:14.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:55:14.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:55:14.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2026-02-22T22:55:14.186 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:14.183+0000 7f6f0841dd80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2026-02-22T22:55:14.188 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:55:14.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:14.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:14.189 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:14.189 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:14.189 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:14.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:14.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T22:55:14.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:55:14.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:55:14.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:14.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:14.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:55:14.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:55:14.220 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:14.221 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:14.264 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:14.265 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:14.265 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:14.265 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:14.265 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:14.265 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:14.266 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:55:14.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:55:14.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:14.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:55:14.314 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:55:14.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=efd3aef8-8e6b-4bba-89ef-d3b20b69fc03 2026-02-22T22:55:14.315 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 efd3aef8-8e6b-4bba-89ef-d3b20b69fc03 2026-02-22T22:55:14.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 efd3aef8-8e6b-4bba-89ef-d3b20b69fc03' 2026-02-22T22:55:14.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:55:14.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSiZtpOj03ExAAJEQbaaJ/qGR9ZszQKg8RnA== 2026-02-22T22:55:14.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSiZtpOj03ExAAJEQbaaJ/qGR9ZszQKg8RnA=="}' 2026-02-22T22:55:14.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new efd3aef8-8e6b-4bba-89ef-d3b20b69fc03 -i td/ceph-helpers/0/new.json 2026-02-22T22:55:14.426 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:55:14.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:55:14.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSiZtpOj03ExAAJEQbaaJ/qGR9ZszQKg8RnA== --osd-uuid efd3aef8-8e6b-4bba-89ef-d3b20b69fc03 2026-02-22T22:55:14.450 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:14.447+0000 7efca960f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:14.451 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:14.449+0000 7efca960f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:14.452 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:14.450+0000 7efca960f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:14.453 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:14.450+0000 7efca960f780 -1 bdev(0x556f281cdc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:55:14.453 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:14.450+0000 7efca960f780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:55:16.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:55:16.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:55:16.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:55:16.574 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:55:16.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:55:16.674 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:55:16.674 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:55:16.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:55:16.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:55:16.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:55:16.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:55:16.691 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:16.689+0000 7f408eed2780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:16.693 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:16.690+0000 7f408eed2780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:16.693 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:16.691+0000 7f408eed2780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:55:16.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:55:16.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:16.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:16.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:17.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:55:17.868 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:55:17.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:17.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:17.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:18.020 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:18.017+0000 7f408eed2780 -1 Falling back to public interface 2026-02-22T22:55:18.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:18.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:18.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:55:18.965 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:55:18.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:18.966 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:19.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:20.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:20.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:20.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:55:20.062 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:55:20.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:20.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:20.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:20.721 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:20.718+0000 7f408eed2780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:55:21.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:21.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:21.161 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:55:21.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:55:21.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:21.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:21.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:22.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:22.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:22.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:55:22.260 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:55:22.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:22.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:22.356 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2635722236,v1:127.0.0.1:6801/2635722236] [v2:127.0.0.1:6802/2635722236,v1:127.0.0.1:6803/2635722236] exists,up efd3aef8-8e6b-4bba-89ef-d3b20b69fc03 2026-02-22T22:55:22.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:55:22.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:55:22.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:55:22.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2026-02-22T22:55:22.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:55:22.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:55:22.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:55:22.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:55:22.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:55:22.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:55:22.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:55:22.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:55:22.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:55:22.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:55:22.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:55:22.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:55:22.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:55:22.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:55:22.363 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:22.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:55:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:55:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:55:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:55:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:55:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:55:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:55:22.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:55:22.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:55:22.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:55:22.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:55:22.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:55:22.366 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:55:22.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd45ac4c-5a36-47c3-b815-879ba6a05ffc 2026-02-22T22:55:22.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bd45ac4c-5a36-47c3-b815-879ba6a05ffc' 2026-02-22T22:55:22.367 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 bd45ac4c-5a36-47c3-b815-879ba6a05ffc 2026-02-22T22:55:22.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:55:22.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaiZtpb/oaFhAAHGTaXGgftcUS58fkYYtsUw== 2026-02-22T22:55:22.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaiZtpb/oaFhAAHGTaXGgftcUS58fkYYtsUw=="}' 2026-02-22T22:55:22.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd45ac4c-5a36-47c3-b815-879ba6a05ffc -i td/ceph-helpers/1/new.json 2026-02-22T22:55:22.472 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:55:22.478 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:55:22.478 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaiZtpb/oaFhAAHGTaXGgftcUS58fkYYtsUw== --osd-uuid bd45ac4c-5a36-47c3-b815-879ba6a05ffc 2026-02-22T22:55:22.495 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:22.492+0000 7fd56201a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:22.496 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:22.494+0000 7fd56201a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:22.498 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:22.494+0000 7fd56201a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:22.498 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:22.495+0000 7fd56201a780 -1 bdev(0x5651a0d13c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:55:22.498 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:22.495+0000 7fd56201a780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:55:24.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:55:24.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:55:24.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:55:24.867 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:55:24.867 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:55:24.967 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:55:24.968 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:55:24.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:55:24.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:55:24.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:55:24.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:55:24.982 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:24.981+0000 7fbf64b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:24.984 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:24.982+0000 7fbf64b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:24.985 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:24.983+0000 7fbf64b19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:25.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:55:25.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:55:25.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:55:25.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:55:25.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:55:25.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:25.068 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:55:25.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:55:25.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:25.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:55:25.164 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:25.824 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:25.822+0000 7fbf64b19780 -1 Falling back to public interface 2026-02-22T22:55:26.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:26.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:26.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:55:26.166 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:55:26.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:26.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:55:26.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:27.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:27.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:27.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:55:27.263 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:55:27.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:27.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:55:27.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:28.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:28.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:28.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:55:28.362 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:55:28.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:28.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:55:28.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:28.513 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:28.511+0000 7fbf64b19780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:55:29.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:29.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:29.460 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:55:29.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:55:29.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:29.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:55:29.559 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3164906521,v1:127.0.0.1:6809/3164906521] [v2:127.0.0.1:6810/3164906521,v1:127.0.0.1:6811/3164906521] exists,up bd45ac4c-5a36-47c3-b815-879ba6a05ffc 2026-02-22T22:55:29.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:55:29.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:29.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2026-02-22T22:55:29.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:55:29.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dbca06c9-b9e3-445b-b9d6-eb622ac06c9b 2026-02-22T22:55:29.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dbca06c9-b9e3-445b-b9d6-eb622ac06c9b' 2026-02-22T22:55:29.562 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 dbca06c9-b9e3-445b-b9d6-eb622ac06c9b 2026-02-22T22:55:29.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:55:29.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhiZtp/n4mIhAA7nXnD3XaMt0F49OykQRUpw== 2026-02-22T22:55:29.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhiZtp/n4mIhAA7nXnD3XaMt0F49OykQRUpw=="}' 2026-02-22T22:55:29.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dbca06c9-b9e3-445b-b9d6-eb622ac06c9b -i td/ceph-helpers/2/new.json 2026-02-22T22:55:29.677 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:55:29.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2026-02-22T22:55:29.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhiZtp/n4mIhAA7nXnD3XaMt0F49OykQRUpw== --osd-uuid dbca06c9-b9e3-445b-b9d6-eb622ac06c9b 2026-02-22T22:55:29.701 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:29.700+0000 7f6aee7f5780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:29.702 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:29.701+0000 7f6aee7f5780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:29.703 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:29.702+0000 7f6aee7f5780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:29.704 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:29.702+0000 7f6aee7f5780 -1 bdev(0x56539966fc00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:55:29.704 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:29.702+0000 7f6aee7f5780 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2026-02-22T22:55:31.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2026-02-22T22:55:31.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:55:31.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:55:31.821 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:55:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:55:31.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:55:31.921 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:55:31.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:55:31.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:55:31.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:55:31.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:55:31.937 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:31.935+0000 7f70c8a0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:31.938 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:31.937+0000 7f70c8a0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:31.939 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:31.938+0000 7f70c8a0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:55:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:55:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:55:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:55:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:55:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:55:32.021 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:55:32.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:32.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:55:32.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:33.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:33.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:33.121 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:55:33.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:55:33.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:33.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:55:33.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:33.276 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:33.274+0000 7f70c8a0f780 -1 Falling back to public interface 2026-02-22T22:55:34.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:34.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:34.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:55:34.217 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:55:34.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:34.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:55:34.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:35.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:35.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:35.316 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:55:35.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:55:35.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:35.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:55:35.413 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:35.987 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:35.985+0000 7f70c8a0f780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:55:36.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:36.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:36.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:55:36.416 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:55:36.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:36.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:55:36.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:37.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:37.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:37.516 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:55:37.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:55:37.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:37.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:55:37.615 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/1672233668,v1:127.0.0.1:6817/1672233668] [v2:127.0.0.1:6818/1672233668,v1:127.0.0.1:6819/1672233668] exists,up dbca06c9-b9e3-445b-b9d6-eb622ac06c9b 2026-02-22T22:55:37.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:55:37.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:55:37.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:55:37.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2026-02-22T22:55:37.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:55:37.708 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:55:37.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:55:37.714 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:55:37.862 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:55:37.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:55:38.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:55:39.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2026-02-22T22:55:39.152 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2026-02-22T22:55:39.250 INFO:tasks.workunit.client.0.trial123.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 22 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2026-02-22T22:55:39.252 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2026-02-22T22:55:39.252 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:39.252 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:39.252 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:39.252 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:39.252 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:39.252 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:39.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2026-02-22T22:55:39.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get osd_pool_default_size 2026-02-22T22:55:39.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2026-02-22T22:55:39.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2026-02-22T22:55:39.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2026-02-22T22:55:39.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2026-02-22T22:55:39.355 INFO:tasks.workunit.client.0.trial123.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2026-02-22T22:55:39.356 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.354+0000 7fb88cafc640 -1 failed for service _ceph-mon._tcp 2026-02-22T22:55:39.356 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.354+0000 7fb88cafc640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2026-02-22T22:55:39.356 INFO:tasks.workunit.client.0.trial123.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2026-02-22T22:55:39.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2026-02-22T22:55:39.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2026-02-22T22:55:39.397 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.395+0000 7f1618a9f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-22T22:55:39.397 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.395+0000 7f1618a9f640 -1 AuthRegistry(0x7f1614069c30) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-22T22:55:39.397 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.395+0000 7f1618a9f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-22T22:55:39.398 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.395+0000 7f1618a9f640 -1 AuthRegistry(0x7f1618a9dff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-22T22:55:39.399 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.396+0000 7f1618a9f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-22T22:55:39.399 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.396+0000 7f1618a9f640 -1 AuthRegistry(0x7f1614066d70) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-22T22:55:39.399 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.396+0000 7f1618a9f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-22T22:55:39.399 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.396+0000 7f1618a9f640 -1 AuthRegistry(0x7f1614106510) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-22T22:55:39.399 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.397+0000 7f1618a9f640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-22T22:55:39.399 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:39.397+0000 7f1618a9f640 -1 AuthRegistry(0x7f1614066d70) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-22T22:55:39.447 INFO:tasks.workunit.client.0.trial123.stdout: cluster: 2026-02-22T22:55:39.447 INFO:tasks.workunit.client.0.trial123.stdout: id: d116af49-5d19-4d5a-91be-c8a20698ede8 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: health: HEALTH_OK 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: services: 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: mon: 1 daemons, quorum a (age 25s) 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: mgr: no daemons active 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: osd: 3 osds: 3 up (since 2s), 3 in (since 9s) 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: data: 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: pools: 0 pools, 0 pgs 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: objects: 0 objects, 0 B 2026-02-22T22:55:39.448 INFO:tasks.workunit.client.0.trial123.stdout: usage: 0 B used, 0 B / 0 B avail 2026-02-22T22:55:39.449 INFO:tasks.workunit.client.0.trial123.stdout: pgs: 2026-02-22T22:55:39.449 INFO:tasks.workunit.client.0.trial123.stdout: 2026-02-22T22:55:39.453 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2026-02-22T22:55:39.453 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:39.454 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:39.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:39.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:39.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:47.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:47.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:55:47.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:55:47.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:55:47.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:55:47.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:55:47.768 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:55:47.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:55:47.784 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:47.782+0000 7f17519dad80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2026-02-22T22:55:47.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:55:47.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:47.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:47.787 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:47.787 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:47.787 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:47.787 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:47.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:55:47.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:55:47.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:55:47.816 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:47.816 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:47.816 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:55:47.816 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:55:47.817 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:47.817 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:47.817 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:47.817 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:47.817 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:47.818 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:47.818 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:47.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:47.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:55:47.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:55:47.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:47.863 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:47.863 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:55:47.863 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:55:47.864 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:47.864 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:47.864 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:47.864 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:47.864 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:47.865 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:47.865 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:47.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:47.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:55:47.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2026-02-22T22:55:47.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:47.911 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:47.911 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:47.911 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:47.911 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:47.912 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:47.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2026-02-22T22:55:47.912 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get osd_pool_default_size 2026-02-22T22:55:47.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2026-02-22T22:55:47.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2026-02-22T22:55:47.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2026-02-22T22:55:47.953 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:47.953 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:47.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:47.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:47.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:48.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:48.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2026-02-22T22:55:48.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2026-02-22T22:55:48.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:55:48.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:55:48.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:55:48.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:55:48.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:55:48.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:55:48.075 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:48.073+0000 7fb0d91ded80 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2026-02-22T22:55:48.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:55:48.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:48.077 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:48.077 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:48.078 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.078 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:48.078 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:55:48.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:55:48.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.107 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:48.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:48.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:55:48.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:55:48.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:48.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:48.151 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:55:48.152 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:48.153 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:55:48.198 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2026-02-22T22:55:48.198 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:48.198 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:48.198 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:48.198 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.198 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.199 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:48.199 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2026-02-22T22:55:48.199 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get osd_pool_default_size 2026-02-22T22:55:48.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2026-02-22T22:55:48.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2026-02-22T22:55:48.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2026-02-22T22:55:48.242 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:48.242 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:48.242 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:48.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:48.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:48.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:48.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2026-02-22T22:55:48.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:55:48.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:55:48.348 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:55:48.348 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:48.348 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:48.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:48.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:48.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:48.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:48.354 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:55:48.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:55:48.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:55:48.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:55:48.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:55:48.356 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:55:48.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:48.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:55:48.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:55:48.358 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:48.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:55:48.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:55:48.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:55:48.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:55:48.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_run_osd td/ceph-helpers 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:55:48.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:55:48.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:55:48.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:55:48.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:55:48.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:55:48.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:55:48.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:55:48.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:55:48.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:55:48.369 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:55:48.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:55:48.369 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:55:48.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:55:48.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:55:48.370 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:55:48.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:48.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:55:48.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:55:48.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:55:48.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:55:48.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:55:48.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:55:48.375 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:55:48.375 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.375 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.375 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:55:48.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:55:48.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:55:48.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:55:48.377 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:55:48.377 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.377 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:55:48.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:55:48.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:55:48.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:55:48.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:55:48.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2026-02-22T22:55:48.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:55:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:55:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:55:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:55:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:55:48.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-22T22:55:48.399 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:55:48.399 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:48.399 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:48.400 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:48.400 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.400 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.400 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:48.400 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:55:48.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:55:48.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:55:48.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:48.428 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:55:48.473 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:55:48.474 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:48.474 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.474 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.474 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:55:48.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:55:48.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:55:48.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2026-02-22T22:55:48.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:55:48.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:55:48.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:55:48.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:55:48.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:55:48.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:55:48.619 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:55:48.619 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:48.619 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:48.620 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:48.620 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.620 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:48.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:55:48.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:48.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:55:48.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:55:48.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:55:48.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:55:48.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:55:48.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:55:48.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:55:48.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4363e0e3-3003-44ce-981e-8726ca57e4c1 2026-02-22T22:55:48.641 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 4363e0e3-3003-44ce-981e-8726ca57e4c1 2026-02-22T22:55:48.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4363e0e3-3003-44ce-981e-8726ca57e4c1' 2026-02-22T22:55:48.641 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:55:48.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0iZtpcz3SJhAA/8oVUdD97rgIkOXACP/aQg== 2026-02-22T22:55:48.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0iZtpcz3SJhAA/8oVUdD97rgIkOXACP/aQg=="}' 2026-02-22T22:55:48.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4363e0e3-3003-44ce-981e-8726ca57e4c1 -i td/ceph-helpers/0/new.json 2026-02-22T22:55:48.757 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:55:48.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:55:48.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0iZtpcz3SJhAA/8oVUdD97rgIkOXACP/aQg== --osd-uuid 4363e0e3-3003-44ce-981e-8726ca57e4c1 2026-02-22T22:55:48.781 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:48.779+0000 7ff2c82e4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:48.783 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:48.780+0000 7ff2c82e4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:48.784 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:48.781+0000 7ff2c82e4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:48.784 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:48.781+0000 7ff2c82e4780 -1 bdev(0x561d7de79c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:55:48.784 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:48.781+0000 7ff2c82e4780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:55:51.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:55:51.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:55:51.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:55:51.671 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:55:51.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:55:51.923 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:55:51.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:55:51.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:55:51.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:55:51.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:55:51.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:55:51.940 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:51.937+0000 7fe8cebb0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:51.941 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:51.939+0000 7fe8cebb0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:51.942 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:51.940+0000 7fe8cebb0780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:52.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:55:52.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:55:52.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:55:52.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:55:52.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:55:52.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:52.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:55:52.115 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:55:52.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:52.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:52.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:53.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:53.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:53.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:55:53.302 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:55:53.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:53.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:53.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:53.526 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:53.523+0000 7fe8cebb0780 -1 Falling back to public interface 2026-02-22T22:55:54.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:54.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:54.491 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:55:54.491 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:55:54.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:54.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:54.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:55.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:55.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:55.683 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:55:55.684 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:55:55.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:55.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:55.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:55:56.230 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:56.227+0000 7fe8cebb0780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:55:56.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:55:56.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:56.876 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:55:56.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:55:56.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:56.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:55:57.065 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3553385722,v1:127.0.0.1:6803/3553385722] [v2:127.0.0.1:6804/3553385722,v1:127.0.0.1:6805/3553385722] exists,up 4363e0e3-3003-44ce-981e-8726ca57e4c1 2026-02-22T22:55:57.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:55:57.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.0.asok 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2026-02-22T22:55:57.068 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.0.asok config get osd_max_backfills 2026-02-22T22:55:57.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2026-02-22T22:55:57.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2026-02-22T22:55:57.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2026-02-22T22:55:57.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2026-02-22T22:55:57.119 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:55:57.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:57.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:55:57.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2026-02-22T22:55:57.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:55:57.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:55:57.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a7bdbf69-9688-4609-b290-3465e7f645dd 2026-02-22T22:55:57.123 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 a7bdbf69-9688-4609-b290-3465e7f645dd 2026-02-22T22:55:57.124 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a7bdbf69-9688-4609-b290-3465e7f645dd' 2026-02-22T22:55:57.124 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:55:57.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9iZtpwYWyBxAAqdkttT0OTw9jpNCP6m4z8w== 2026-02-22T22:55:57.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9iZtpwYWyBxAAqdkttT0OTw9jpNCP6m4z8w=="}' 2026-02-22T22:55:57.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a7bdbf69-9688-4609-b290-3465e7f645dd -i td/ceph-helpers/1/new.json 2026-02-22T22:55:57.323 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:55:57.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:55:57.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQB9iZtpwYWyBxAAqdkttT0OTw9jpNCP6m4z8w== --osd-uuid a7bdbf69-9688-4609-b290-3465e7f645dd 2026-02-22T22:55:57.349 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:57.347+0000 7f3169811780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:57.351 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:57.348+0000 7f3169811780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:57.352 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:57.349+0000 7f3169811780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:57.352 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:57.349+0000 7f3169811780 -1 bdev(0x55e8c4625c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:55:57.352 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:57.350+0000 7f3169811780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:55:59.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:55:59.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:55:59.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:55:59.472 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:55:59.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:55:59.727 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:55:59.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:55:59.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-22T22:55:59.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:55:59.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:55:59.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:55:59.745 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:59.743+0000 7fb248f16780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:59.746 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:59.744+0000 7fb248f16780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:59.747 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:55:59.745+0000 7fb248f16780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:55:59.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:55:59.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:55:59.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:00.103 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:01.077 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:01.075+0000 7fb248f16780 -1 Falling back to public interface 2026-02-22T22:56:01.104 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:01.104 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:01.105 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:56:01.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:56:01.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:01.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:01.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:02.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:02.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:02.296 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:56:02.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:56:02.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:02.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:02.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:03.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:03.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:03.485 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:56:03.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:56:03.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:03.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:03.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:03.782 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:03.781+0000 7fb248f16780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:56:04.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:04.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:04.677 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:56:04.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:56:04.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:04.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:04.869 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/688341160,v1:127.0.0.1:6811/688341160] [v2:127.0.0.1:6812/688341160,v1:127.0.0.1:6813/688341160] exists,up a7bdbf69-9688-4609-b290-3465e7f645dd 2026-02-22T22:56:04.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:56:04.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:56:04.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.1.asok 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2026-02-22T22:56:04.871 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.1.asok config get osd_max_backfills 2026-02-22T22:56:04.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2026-02-22T22:56:04.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:04.921 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2026-02-22T22:56:04.922 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:56:04.923 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8d1049b-d9af-44ce-b461-9580884cfb06 2026-02-22T22:56:04.923 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c8d1049b-d9af-44ce-b461-9580884cfb06' 2026-02-22T22:56:04.923 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 c8d1049b-d9af-44ce-b461-9580884cfb06 2026-02-22T22:56:04.923 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:56:04.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEiZtpB2CQNxAA+DkYVurVd+VlDe7QX9y63Q== 2026-02-22T22:56:04.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEiZtpB2CQNxAA+DkYVurVd+VlDe7QX9y63Q=="}' 2026-02-22T22:56:04.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8d1049b-d9af-44ce-b461-9580884cfb06 -i td/ceph-helpers/2/new.json 2026-02-22T22:56:05.129 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:56:05.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2026-02-22T22:56:05.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEiZtpB2CQNxAA+DkYVurVd+VlDe7QX9y63Q== --osd-uuid c8d1049b-d9af-44ce-b461-9580884cfb06 2026-02-22T22:56:05.157 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:05.155+0000 7fa48342f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:05.158 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:05.157+0000 7fa48342f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:05.159 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:05.158+0000 7fa48342f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:05.160 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:05.158+0000 7fa48342f780 -1 bdev(0x5605b5207c00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:56:05.160 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:05.158+0000 7fa48342f780 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2026-02-22T22:56:07.280 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2026-02-22T22:56:07.280 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:56:07.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:56:07.281 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:56:07.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:56:07.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:56:07.539 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:56:07.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:56:07.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:56:07.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:56:07.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:56:07.556 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:07.554+0000 7fa254e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:07.557 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:07.556+0000 7fa254e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:07.558 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:07.556+0000 7fa254e2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:07.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:56:07.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:56:07.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:56:07.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:56:07.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:56:07.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:07.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:56:07.731 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:56:07.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:07.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:56:07.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:08.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:08.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:08.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:56:08.921 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:56:08.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:08.922 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:56:09.113 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:09.140 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:09.138+0000 7fa254e2c780 -1 Falling back to public interface 2026-02-22T22:56:10.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:10.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:10.115 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:56:10.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:56:10.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:10.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:56:10.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:11.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:11.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:11.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:56:11.309 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:56:11.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:11.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:56:11.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:12.091 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:12.089+0000 7fa254e2c780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:56:12.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:12.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:12.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:56:12.500 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:56:12.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:12.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:56:12.693 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:13.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:13.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:13.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:56:13.695 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:56:13.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:13.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:56:13.884 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/385001694,v1:127.0.0.1:6819/385001694] [v2:127.0.0.1:6820/385001694,v1:127.0.0.1:6821/385001694] exists,up c8d1049b-d9af-44ce-b461-9580884cfb06 2026-02-22T22:56:13.885 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-osd.2.asok 2026-02-22T22:56:13.886 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2026-02-22T22:56:13.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.83985/ceph-osd.2.asok config get osd_max_backfills 2026-02-22T22:56:13.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2026-02-22T22:56:13.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2026-02-22T22:56:13.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2026-02-22T22:56:13.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:13.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:13.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:13.935 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:13.935 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:13.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:13.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:13.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.043 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.045 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.046 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.053 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.053 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.053 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.053 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_set_config td/ceph-helpers 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:14.056 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:14.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:14.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:14.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.058 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.058 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.059 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.060 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.063 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.063 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.063 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.064 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.064 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.064 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:56:14.065 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:56:14.066 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.066 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.066 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:56:14.066 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:56:14.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:56:14.087 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:56:14.088 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:14.088 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:14.088 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:14.088 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.088 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.088 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:14.088 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:56:14.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:14.116 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:14.117 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:14.117 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.117 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.117 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:14.117 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:14.117 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:56:14.161 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:56:14.161 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:14.161 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:14.161 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:56:14.162 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:14.163 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:14.164 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:56:14.206 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2026-02-22T22:56:14.207 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:14.207 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:14.207 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2026-02-22T22:56:14.207 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2026-02-22T22:56:14.208 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:14.208 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:14.208 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:14.209 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:14.209 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.209 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.209 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:14.209 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:14.209 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get ms_crc_header 2026-02-22T22:56:14.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2026-02-22T22:56:14.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2026-02-22T22:56:14.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2026-02-22T22:56:14.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2026-02-22T22:56:14.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2026-02-22T22:56:14.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2026-02-22T22:56:14.251 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:56:14.251 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2026-02-22T22:56:14.251 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:14.251 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:14.252 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:14.252 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.252 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.252 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:14.252 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config set ms_crc_header false 2026-02-22T22:56:14.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:56:14.298 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2026-02-22T22:56:14.298 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:14.298 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:14.298 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2026-02-22T22:56:14.298 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2026-02-22T22:56:14.298 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:14.298 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:14.299 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:14.299 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:14.299 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.299 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.299 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:14.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:14.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get ms_crc_header 2026-02-22T22:56:14.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2026-02-22T22:56:14.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2026-02-22T22:56:14.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2026-02-22T22:56:14.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2026-02-22T22:56:14.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2026-02-22T22:56:14.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:56:14.341 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:56:14.341 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2026-02-22T22:56:14.341 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:14.341 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:14.341 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:14.341 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.341 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.342 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:14.342 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config set ms_crc_header true 2026-02-22T22:56:14.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:56:14.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2026-02-22T22:56:14.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:14.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:14.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2026-02-22T22:56:14.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2026-02-22T22:56:14.384 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:14.384 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:14.384 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:14.385 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:14.385 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.385 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.385 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:14.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:14.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get ms_crc_header 2026-02-22T22:56:14.428 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:14.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.535 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.535 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.536 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.537 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.538 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.539 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.541 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.542 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.542 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.543 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.543 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_setup td/ceph-helpers 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:14.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:14.546 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:14.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:14.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:14.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.547 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.547 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.550 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.551 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.551 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.552 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.552 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.552 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:56:14.554 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:56:14.555 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.555 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:56:14.555 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:56:14.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:56:14.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:56:14.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:14.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:14.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.559 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.560 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.561 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.565 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:56:14.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:56:14.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.567 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:56:14.568 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:14.570 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:14.571 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:14.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:14.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:14.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.574 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.575 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.575 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.576 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.576 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.577 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_teardown td/ceph-helpers 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:14.580 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:14.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:14.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.581 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.581 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.582 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.582 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.583 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.587 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.587 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.587 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:56:14.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:56:14.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:56:14.590 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:56:14.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:14.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:14.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:14.593 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:14.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:14.594 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:14.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:14.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:14.596 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:14.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:14.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:14.597 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:14.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:14.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:14.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:14.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:14.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:14.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:14.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:14.602 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:14.602 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_wait_background td/ceph-helpers 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_wait_background: local pids= 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: run_in_background: local pid_variable=pids 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: run_in_background: shift 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: eval 'pids+=" 151837"' 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: pids+=' 151837' 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2026-02-22T22:56:14.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: run_in_background: local pid_variable=pids 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: run_in_background: shift 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: eval 'pids+=" 151838"' 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: pids+=' 151838' 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: test_wait_background: wait_background pids 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: wait_background: pids=' 151837 151838' 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: wait_background: return_code=0 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: wait_background: for pid in $pids 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: wait 151837 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: bash -c 'sleep 1; exit 1' 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sed 's/^/151840: /' 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: bash -c 'sleep 2; exit 0' 2026-02-22T22:56:14.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sed 's/^/151842: /' 2026-02-22T22:56:15.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: return 1 2026-02-22T22:56:15.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return_code=1 2026-02-22T22:56:15.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: wait_background: for pid in $pids 2026-02-22T22:56:15.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: wait 151838 2026-02-22T22:56:16.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: return 0 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: wait_background: eval 'pids='\'''\''' 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: wait_background: pids= 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: wait_background: return 1 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: test_wait_background: '[' 1 -ne 1 ']' 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: run_in_background: local pid_variable=pids 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: run_in_background: shift 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: eval 'pids+=" 151845"' 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: pids+=' 151845' 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: run_in_background: local pid_variable=pids 2026-02-22T22:56:16.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: run_in_background: shift 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: eval 'pids+=" 151846"' 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2195: run_in_background: pids+=' 151846' 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: test_wait_background: wait_background pids 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: wait_background: pids=' 151845 151846' 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: wait_background: return_code=0 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: wait_background: for pid in $pids 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: wait 151845 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: bash -c 'sleep 1; exit 0' 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sed 's/^/151849: /' 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: bash -c 'sleep 2; exit 0' 2026-02-22T22:56:16.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: sed 's/^/151850: /' 2026-02-22T22:56:17.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: return 0 2026-02-22T22:56:17.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: wait_background: for pid in $pids 2026-02-22T22:56:17.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: wait 151846 2026-02-22T22:56:18.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: run_in_background: return 0 2026-02-22T22:56:18.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: wait_background: eval 'pids='\'''\''' 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: wait_background: pids= 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: wait_background: return 0 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: test_wait_background: '[' 0 -ne 0 ']' 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: test_wait_background: '[' '!' -z '' ']' 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_wait_for_clean td/ceph-helpers 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1691: test_wait_for_clean: local dir=td/ceph-helpers 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1693: test_wait_for_clean: setup td/ceph-helpers 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:18.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:18.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:18.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:18.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:18.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:18.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:18.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:18.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:18.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:18.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:18.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:18.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:18.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:18.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:18.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:18.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:18.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:18.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:18.617 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:18.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:18.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:18.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:18.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:18.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:18.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:56:18.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:56:18.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:18.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:18.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:56:18.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:56:18.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:56:18.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:56:18.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:56:18.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1694: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-22T22:56:18.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:56:18.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:56:18.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:56:18.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:56:18.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:56:18.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-22T22:56:18.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:56:18.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:18.644 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:18.644 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:18.644 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:18.644 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:18.645 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:18.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T22:56:18.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:56:18.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:56:18.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:18.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:18.672 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:18.673 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:18.673 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:56:18.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:56:18.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:18.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:18.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:56:18.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:56:18.719 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:18.719 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:18.719 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:18.720 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:18.720 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:18.720 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:18.720 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:18.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:18.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:56:18.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1695: test_wait_for_clean: run_osd td/ceph-helpers 0 2026-02-22T22:56:18.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:56:18.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:56:18.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:56:18.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:18.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:56:18.765 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:56:18.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b7fa79d-ee3a-47c1-ac59-85bc369e4d06 2026-02-22T22:56:18.766 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b7fa79d-ee3a-47c1-ac59-85bc369e4d06' 2026-02-22T22:56:18.766 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 6b7fa79d-ee3a-47c1-ac59-85bc369e4d06 2026-02-22T22:56:18.766 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:56:18.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSiZtpLaogLhAAbeOFAXRKFiFwULUPD8VUPQ== 2026-02-22T22:56:18.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSiZtpLaogLhAAbeOFAXRKFiFwULUPD8VUPQ=="}' 2026-02-22T22:56:18.776 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b7fa79d-ee3a-47c1-ac59-85bc369e4d06 -i td/ceph-helpers/0/new.json 2026-02-22T22:56:18.877 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:56:18.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:56:18.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSiZtpLaogLhAAbeOFAXRKFiFwULUPD8VUPQ== --osd-uuid 6b7fa79d-ee3a-47c1-ac59-85bc369e4d06 2026-02-22T22:56:18.900 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:18.898+0000 7fd69a53f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:18.901 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:18.900+0000 7fd69a53f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:18.902 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:18.901+0000 7fd69a53f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:18.903 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:18.901+0000 7fd69a53f780 -1 bdev(0x559e40a1dc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:56:18.903 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:18.901+0000 7fd69a53f780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:56:21.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:56:21.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:56:21.026 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:56:21.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:56:21.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:56:21.129 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:56:21.129 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:56:21.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:56:21.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:56:21.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:56:21.130 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:56:21.145 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:21.144+0000 7f7b89423780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:21.147 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:21.145+0000 7f7b89423780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:21.148 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:21.146+0000 7f7b89423780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:21.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:56:21.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:56:21.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:56:21.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:56:21.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:56:21.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:21.227 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:56:21.227 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:56:21.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:21.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:21.324 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:22.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:22.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:22.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:56:22.327 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:56:22.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:22.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:22.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:22.481 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:22.479+0000 7f7b89423780 -1 Falling back to public interface 2026-02-22T22:56:23.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:23.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:23.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:56:23.425 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:56:23.426 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:23.426 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:23.521 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:24.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:24.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:24.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:56:24.523 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:56:24.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:24.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:24.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:25.170 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:25.167+0000 7f7b89423780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:56:25.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:25.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:25.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:56:25.623 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:56:25.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:25.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:25.724 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:26.725 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:26.726 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:26.726 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:56:26.726 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:56:26.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:26.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:26.822 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/720119218,v1:127.0.0.1:6801/720119218] [v2:127.0.0.1:6802/720119218,v1:127.0.0.1:6803/720119218] exists,up 6b7fa79d-ee3a-47c1-ac59-85bc369e4d06 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1696: test_wait_for_clean: run_mgr td/ceph-helpers x 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:56:26.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:56:26.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:56:26.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:56:26.924 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:56:26.924 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:26.924 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:26.924 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:26.924 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:26.924 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:26.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:26.925 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:56:26.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:56:26.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1697: test_wait_for_clean: create_rbd_pool 2026-02-22T22:56:26.942 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:56:27.036 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:56:27.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:56:27.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:56:27.222 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:56:27.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:56:28.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:56:28.531 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1698: test_wait_for_clean: wait_for_clean 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 1 .1 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:56:28.532 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:56:28.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:56:28.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2026-02-22T22:56:28.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:56:28.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:56:28.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:56:28.556 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:56:28.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:56:28.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:56:28.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:56:28.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:56:28.656 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:56:28.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=0 2026-02-22T22:56:28.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 0 2026-02-22T22:56:28.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-0' 2026-02-22T22:56:28.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:56:28.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-0 2026-02-22T22:56:28.720 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:56:28.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:56:28.721 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-0 2026-02-22T22:56:28.721 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:56:28.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=0 2026-02-22T22:56:28.722 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 0' 2026-02-22T22:56:28.722 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 0 2026-02-22T22:56:28.723 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:56:28.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 0 -lt 0 2026-02-22T22:56:28.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:56:28.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:28.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:28.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 == 0 2026-02-22T22:56:28.924 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: sleep 1 2026-02-22T22:56:29.926 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:56:29.927 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:29.927 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:30.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:56:30.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:30.180 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:30.180 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:30.180 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:30.181 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:30.181 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:30.181 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:30.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:30.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:30.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:30.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:30.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:30.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' -1 2026-02-22T22:56:30.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: loop=0 2026-02-22T22:56:30.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: num_active_clean=0 2026-02-22T22:56:30.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:30.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:56:30.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:30.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:30.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:30.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:30.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:30.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:30.718 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:30.719 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:30.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:30.905 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:30.906 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:30.906 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:31.155 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:31.156 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:31.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:56:31.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:56:31.404 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 1 >= 4 )) 2026-02-22T22:56:31.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:31.405 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.2 2026-02-22T22:56:31.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:31.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:31.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:31.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:31.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:31.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:31.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:31.608 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:31.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:31.792 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:31.793 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:31.793 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:32.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:32.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:56:32.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:32.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:32.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:32.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:32.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:32.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:32.042 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:32.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:56:32.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:56:32.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 2 >= 4 )) 2026-02-22T22:56:32.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:32.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.4 2026-02-22T22:56:32.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:32.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:32.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:32.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:32.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:32.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:32.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:32.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:32.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:32.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:32.887 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:32.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:33.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:33.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:56:33.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:33.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:33.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:33.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:33.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:33.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:33.139 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:33.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:56:33.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:56:33.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 3 >= 4 )) 2026-02-22T22:56:33.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:33.390 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.3 2026-02-22T22:56:33.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:33.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:33.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:33.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:33.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:33.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:33.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:33.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:33.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:33.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:33.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:33.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:34.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:34.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:56:34.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:34.131 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:34.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:34.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:34.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:34.132 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:34.132 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:34.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:56:34.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:56:34.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 4 >= 4 )) 2026-02-22T22:56:34.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: wait_for_clean: ceph report 2026-02-22T22:56:34.629 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:56:34.629 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_fingerprint": "8c9bfb03-4449-4376-aff9-b2cd7b280457", 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "version": "19.2.3-689-gf90f4eae", 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "commit": "f90f4eaebb28b619cf76e80cbbe997656da69579", 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "timestamp": "2026-02-22T22:56:34.627598+0000", 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "tag": "", 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "health": { 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "status": "HEALTH_WARN", 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "checks": { 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "PG_DEGRADED": { 2026-02-22T22:56:34.630 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: "count": 2 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [ 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.631 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "TOO_FEW_OSDS": { 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "count": 2 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [], 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.632 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "mutes": [] 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "monmap_first_committed": 1, 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "monmap_last_committed": 1, 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "monmap": { 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "fsid": "d116af49-5d19-4d5a-91be-c8a20698ede8", 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:56:18.635577Z", 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "created": "2026-02-22T22:56:18.635577Z", 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "min_mon_release": 19, 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "min_mon_release_name": "squid", 2026-02-22T22:56:34.633 INFO:tasks.workunit.client.0.trial123.stdout: "election_strategy": 1, 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "disallowed_leaders": "", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode": false, 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "tiebreaker_mon": "", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "removed_ranks": "", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "features": { 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "persistent": [ 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "kraken", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "luminous", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "mimic", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap-prune", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "nautilus", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "octopus", 2026-02-22T22:56:34.634 INFO:tasks.workunit.client.0.trial123.stdout: "pacific", 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "elector-pinging", 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "quincy", 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "reef", 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "squid" 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "optional": [] 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "mons": [ 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "rank": 0, 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "name": "a", 2026-02-22T22:56:34.635 INFO:tasks.workunit.client.0.trial123.stdout: "public_addrs": { 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:7109", 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 0 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:7109/0", 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "public_addr": "127.0.0.1:7109/0", 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "priority": 0, 2026-02-22T22:56:34.636 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 0, 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "crush_location": "{}" 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "quorum": [ 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: 0 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap": { 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 12, 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "fsid": "d116af49-5d19-4d5a-91be-c8a20698ede8", 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "created": "2026-02-22T22:56:18.669496+0000", 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:56:28.524058+0000", 2026-02-22T22:56:34.637 INFO:tasks.workunit.client.0.trial123.stdout: "last_up_change": "2026-02-22T22:56:25.766644+0000", 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "last_in_change": "2026-02-22T22:56:18.853786+0000", 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "flags_num": 5799936, 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "flags_set": [ 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "pglog_hardlimit", 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snapdirs", 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "recovery_deletes", 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "sortbitwise" 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "crush_version": 3, 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "full_ratio": 0.99000000953674316, 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "backfillfull_ratio": 0.99000000953674316, 2026-02-22T22:56:34.638 INFO:tasks.workunit.client.0.trial123.stdout: "nearfull_ratio": 0.85000002384185791, 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_snapshot": "", 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "pool_max": 1, 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "max_osd": 1, 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "require_min_compat_client": "luminous", 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "min_compat_client": "jewel", 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "require_osd_release": "squid", 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "allow_crimson": false, 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "pools": [ 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "pool_name": "rbd", 2026-02-22T22:56:34.639 INFO:tasks.workunit.client.0.trial123.stdout: "create_time": "2026-02-22T22:56:27.121987+0000", 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "flags": 8193, 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "type": 1, 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "size": 2, 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "min_size": 1, 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "crush_rule": 0, 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_count": 0, 2026-02-22T22:56:34.640 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_target": 0, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_barrier": 0, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "is_stretch_pool": false, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "object_hash": 2, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "pg_autoscale_mode": "off", 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num": 4, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "pg_placement_num": 4, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "pg_placement_num_target": 4, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num_target": 4, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num_pending": 4, 2026-02-22T22:56:34.641 INFO:tasks.workunit.client.0.trial123.stdout: "last_pg_merge_meta": { 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "source_pgid": "0.0", 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "ready_epoch": 0, 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_started": 0, 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_clean": 0, 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "source_version": "0'0", 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "target_version": "0'0" 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "last_change": "12", 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend": "0", 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend_prenautilus": "0", 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend_preluminous": "0", 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "auid": 0, 2026-02-22T22:56:34.642 INFO:tasks.workunit.client.0.trial123.stdout: "snap_mode": "selfmanaged", 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "snap_seq": 2, 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "snap_epoch": 12, 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "pool_snaps": [], 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "removed_snaps": "[]", 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "quota_max_bytes": 0, 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "quota_max_objects": 0, 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "tiers": [], 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "tier_of": -1, 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "read_tier": -1, 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "write_tier": -1, 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "cache_mode": "none", 2026-02-22T22:56:34.643 INFO:tasks.workunit.client.0.trial123.stdout: "target_max_bytes": 0, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "target_max_objects": 0, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_dirty_ratio_micro": 400000, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_full_ratio_micro": 800000, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "cache_min_flush_age": 0, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "cache_min_evict_age": 0, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "erasure_code_profile": "", 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_params": { 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "type": "none" 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_period": 0, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_count": 0, 2026-02-22T22:56:34.644 INFO:tasks.workunit.client.0.trial123.stdout: "use_gmt_hitset": true, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "min_read_recency_for_promote": 0, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "min_write_recency_for_promote": 0, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_grade_decay_rate": 0, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_search_last_n": 0, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "grade_table": [], 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "stripe_width": 0, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "expected_num_objects": 0, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "fast_read": false, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "options": {}, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "application_metadata": { 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "rbd": {} 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.645 INFO:tasks.workunit.client.0.trial123.stdout: "read_balance": { 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "score_type": "Fair distribution", 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "score_acting": 1, 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "score_stable": 1, 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "optimal_score": 1, 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "raw_score_acting": 1, 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "raw_score_stable": 1, 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "primary_affinity_weighted": 1, 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "average_primary_affinity": 1, 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "average_primary_affinity_weighted": 1 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.646 INFO:tasks.workunit.client.0.trial123.stdout: "osds": [ 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "uuid": "6b7fa79d-ee3a-47c1-ac59-85bc369e4d06", 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "up": 1, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "in": 1, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 1, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "primary_affinity": 1, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "last_clean_begin": 0, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "last_clean_end": 0, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 5, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "up_thru": 7, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "down_at": 0, 2026-02-22T22:56:34.647 INFO:tasks.workunit.client.0.trial123.stdout: "lost_at": 0, 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "public_addrs": { 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6800", 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6801", 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.648 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_addrs": { 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6802", 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:56:34.649 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6803", 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_back_addrs": { 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6806", 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:56:34.650 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6807", 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_front_addrs": { 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6804", 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.651 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6805", 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 720119218 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "public_addr": "127.0.0.1:6801/720119218", 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_addr": "127.0.0.1:6803/720119218", 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_back_addr": "127.0.0.1:6807/720119218", 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_front_addr": "127.0.0.1:6805/720119218", 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "state": [ 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "exists", 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: "up" 2026-02-22T22:56:34.652 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "osd_xinfo": [ 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "down_stamp": "0.000000", 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "laggy_probability": 0, 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "laggy_interval": 0, 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "features": 4540701547738038271, 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "old_weight": 0, 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "last_purged_snaps_scrub": "2026-02-22T22:56:26.152152+0000", 2026-02-22T22:56:34.653 INFO:tasks.workunit.client.0.trial123.stdout: "dead_epoch": 0 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap": [], 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap_items": [], 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap_primaries": [], 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "pg_temp": [], 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "primary_temp": [], 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "blocklist": {}, 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "range_blocklist": {}, 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "erasure_code_profiles": { 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "default": { 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "k": "2", 2026-02-22T22:56:34.654 INFO:tasks.workunit.client.0.trial123.stdout: "m": "2", 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: "plugin": "jerasure", 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: "technique": "reed_sol_van" 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: "removed_snaps_queue": [ 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: "snaps": [ 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: "begin": 2, 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: "length": 1 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.655 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: "new_removed_snaps": [ 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: "snaps": [ 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: "begin": 2, 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: "length": 1 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.656 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "new_purged_snaps": [], 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "crush_node_flags": {}, 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "device_class_flags": {}, 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode": { 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode_enabled": false, 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_bucket_count": 0, 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "degraded_stretch_mode": 0, 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "recovering_stretch_mode": 0, 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode_bucket": 0 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: "osd_metadata": [ 2026-02-22T22:56:34.657 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "arch": "x86_64", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "back_addr": "[v2:127.0.0.1:6802/720119218,v1:127.0.0.1:6803/720119218]", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "back_iface": "", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs": "1", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_dedicated_db": "0", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_dedicated_wal": "0", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_single_shared_device": "1", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_allocation_from_file": "1", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_allocator": "hybrid", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_access_mode": "file", 2026-02-22T22:56:34.658 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_block_size": "4096", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_devices": "nvme1n1", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_driver": "KernelDevice", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_optimal_io_size": "4096", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_rotational": "0", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_size": "107374182400", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_support_discard": "1", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_type": "ssd", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_min_alloc_size": "4096", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_release": "squid", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version": "ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable)", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version_short": "19.2.3-689-gf90f4eae", 2026-02-22T22:56:34.659 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version_when_created": "ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable)", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "cpu": "AMD EPYC 4464P 12-Core Processor", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "created_at": "2026-02-22T22:56:20.507898Z", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "default_device_class": "ssd", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "device_ids": "nvme1n1=Micron_7450_MTFDKBA960TFR_25174FCD7341", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "device_paths": "nvme1n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "devices": "nvme1n1", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "distro": "centos", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "distro_description": "CentOS Stream 9", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "distro_version": "9", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "front_addr": "[v2:127.0.0.1:6800/720119218,v1:127.0.0.1:6801/720119218]", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "front_iface": "", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/720119218,v1:127.0.0.1:6807/720119218]", 2026-02-22T22:56:34.660 INFO:tasks.workunit.client.0.trial123.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/720119218,v1:127.0.0.1:6805/720119218]", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "hostname": "trial123", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "journal_rotational": "0", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Feb 11 20:19:22 UTC 2026", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "kernel_version": "5.14.0-681.el9.x86_64", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "mem_swap_kb": "8388604", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "mem_total_kb": "130438940", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "objectstore_numa_node": "0", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "objectstore_numa_nodes": "0", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "os": "Linux", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "osd_data": "td/ceph-helpers/0", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "osd_objectstore": "bluestore", 2026-02-22T22:56:34.661 INFO:tasks.workunit.client.0.trial123.stdout: "osdspec_affinity": "", 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "rotational": "0" 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_clean_epochs": { 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "min_last_epoch_clean": 0, 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_clean": { 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "per_pool": [] 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "osd_epochs": [ 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 5 2026-02-22T22:56:34.662 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_first_committed": 1, 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_last_committed": 12, 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "crushmap": { 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "devices": [ 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "name": "osd.0", 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "class": "ssd" 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.663 INFO:tasks.workunit.client.0.trial123.stdout: "types": [ 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 0, 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "name": "osd" 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "name": "host" 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 2, 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: "name": "chassis" 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.664 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 3, 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "name": "rack" 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 4, 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "name": "row" 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 5, 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "name": "pdu" 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 6, 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: "name": "pod" 2026-02-22T22:56:34.665 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 7, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "name": "room" 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 8, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "name": "datacenter" 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 9, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "name": "zone" 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 10, 2026-02-22T22:56:34.666 INFO:tasks.workunit.client.0.trial123.stdout: "name": "region" 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "name": "root" 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "buckets": [ 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "id": -1, 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "name": "default", 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "root", 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:56:34.667 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "id": -3, 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "id": -2, 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "name": "default~ssd", 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "root", 2026-02-22T22:56:34.668 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "id": -4, 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "id": -3, 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "name": "trial123", 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "host", 2026-02-22T22:56:34.669 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "id": -4, 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "name": "trial123~ssd", 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "host", 2026-02-22T22:56:34.670 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "rules": [ 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "rule_id": 0, 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "rule_name": "replicated_rule", 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "type": 1, 2026-02-22T22:56:34.671 INFO:tasks.workunit.client.0.trial123.stdout: "steps": [ 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "op": "take", 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "item": -1, 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "item_name": "default" 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "op": "choose_firstn", 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "num": 0, 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "type": "osd" 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "op": "emit" 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.672 INFO:tasks.workunit.client.0.trial123.stdout: "tunables": { 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "choose_local_tries": 0, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "choose_local_fallback_tries": 0, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "choose_total_tries": 50, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_descend_once": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_vary_r": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_stable": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "msr_descents": 100, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "msr_collision_tries": 100, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "straw_calc_version": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "allowed_bucket_algs": 54, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "profile": "jewel", 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "optimal_tunables": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "legacy_tunables": 0, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "minimum_required_version": "jewel", 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables2": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "has_v2_rules": 0, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables3": 1, 2026-02-22T22:56:34.673 INFO:tasks.workunit.client.0.trial123.stdout: "has_v3_rules": 0, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "has_v4_buckets": 1, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables5": 1, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "has_v5_rules": 0, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "has_msr_rules": 0 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "choose_args": {} 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "fsmap": { 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "btime": "2026-02-22T22:56:18:669344+0000", 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "default_fscid": -1, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "compat": { 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "compat": {}, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "ro_compat": {}, 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "incompat": { 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "feature_1": "base v0.20", 2026-02-22T22:56:34.674 INFO:tasks.workunit.client.0.trial123.stdout: "feature_2": "client writeable ranges", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_3": "default file layouts on dirs", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_4": "dir inode in separate object", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_5": "mds uses versioned encoding", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_6": "dirfrag is stored in omap", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_8": "no anchor table", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_9": "file layout v2", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_10": "snaprealm v2", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_11": "minor log segments", 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_12": "quiesce subvolumes" 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "feature_flags": { 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "enable_multiple": true, 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "ever_enabled_multiple": true 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "standbys": [], 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "filesystems": [] 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.675 INFO:tasks.workunit.client.0.trial123.stdout: "mdsmap_first_committed": 1, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "mdsmap_last_committed": 1, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "auth": { 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "first_committed": 1, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "last_committed": 3, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_secrets": 1 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg": 4, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_active": 4, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_unknown": 0, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_osd": 1, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "pool_sum": { 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "stat_sum": { 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes": 19, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects": 2, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_clones": 0, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_copies": 4, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing": 0, 2026-02-22T22:56:34.676 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_degraded": 2, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_misplaced": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_unfound": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_dirty": 2, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_whiteouts": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_read": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_read_kb": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_write": 2, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_write_kb": 2, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_scrub_errors": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_recovered": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_recovered": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_keys_recovered": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_omap": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_kb": 0, 2026-02-22T22:56:34.677 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_kb": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_promote": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_high": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_low": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_some": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_full": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_pinned": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_large_omap_objects": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_manifest": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_bytes": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_keys": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_repaired": 0 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "store_stats": { 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "total": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "available": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 0, 2026-02-22T22:56:34.678 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 0 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "log_size": 3, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "ondisk_log_size": 3, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "up": 4, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "acting": 4, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "num_store_stats": 0 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "osd_sum": { 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "num_pgs": 0, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "num_osds": 1, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_osds": 1, 2026-02-22T22:56:34.679 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "kb": 104857600, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used": 27060, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_data": 80, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_omap": 1, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_meta": 26942, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "kb_avail": 104830540, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "statfs": { 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "total": 107374182400, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "available": 107346472960, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 81920, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 16230, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 1593, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 27589063 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "hb_peers": [], 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "snap_trim_queue_len": 0, 2026-02-22T22:56:34.680 INFO:tasks.workunit.client.0.trial123.stdout: "num_snap_trimming": 0, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "num_shards_repaired": 0, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "op_queue_age_hist": { 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "histogram": [], 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "upper_bound": 1 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "perf_stat": { 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ms": 1, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ms": 1, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ns": 1000000, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ns": 1000000 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "alerts": [], 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "network_ping_times": [] 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "osd_sum_by_class": { 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "ssd": { 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 0, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 0, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "num_pgs": 0, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "num_osds": 1, 2026-02-22T22:56:34.681 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_osds": 1, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "kb": 104857600, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used": 27060, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_data": 80, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_omap": 1, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_meta": 26942, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "kb_avail": 104830540, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "statfs": { 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "total": 107374182400, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "available": 107346472960, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 81920, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 16230, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 1593, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 27589063 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "hb_peers": [], 2026-02-22T22:56:34.682 INFO:tasks.workunit.client.0.trial123.stdout: "snap_trim_queue_len": 0, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "num_snap_trimming": 0, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "num_shards_repaired": 0, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "op_queue_age_hist": { 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "histogram": [], 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "upper_bound": 1 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "perf_stat": { 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ms": 1, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ms": 1, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ns": 1000000, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ns": 1000000 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "alerts": [], 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "network_ping_times": [] 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "pool_stats": [ 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "poolid": 1, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg": 4, 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "stat_sum": { 2026-02-22T22:56:34.683 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes": 19, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects": 2, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_clones": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_copies": 4, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_degraded": 2, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_misplaced": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_unfound": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_dirty": 2, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_whiteouts": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_read": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_read_kb": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_write": 2, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_write_kb": 2, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_scrub_errors": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_recovered": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_recovered": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_keys_recovered": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_omap": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:56:34.684 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_kb": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_kb": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_promote": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_high": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_low": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_some": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_full": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_pinned": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_large_omap_objects": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_manifest": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_bytes": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_keys": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_repaired": 0 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "store_stats": { 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "total": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "available": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 4096, 2026-02-22T22:56:34.685 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 19, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 0, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 0 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "log_size": 3, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "ondisk_log_size": 3, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "up": 4, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "acting": 4, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "num_store_stats": 1 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "osd_stats": [ 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 21474836481 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_by_state": [ 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "state": "active+undersized+degraded", 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: "num": 2 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.686 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "state": "active+undersized", 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "num": 2 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_by_osd": [ 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "num_primary_pg": 4, 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "num_acting_pg": 4, 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "num_up_not_acting_pg": 0 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snaps": [ 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snaps": {} 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "servicemap": { 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:56:18.669619+0000", 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "services": {} 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:56:34.687 INFO:tasks.workunit.client.0.trial123.stdout: "mgrstat_first_committed": 1, 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "mgrstat_last_committed": 4, 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "logm_first_committed": 1, 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "logm_last_committed": 58, 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "paxos": { 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "first_committed": 1, 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "last_committed": 83, 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "last_pn": 0, 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: "accepted_pn": 0 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:56:34.688 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:report 909345569 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: wait_for_clean: return 1 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: test_wait_for_clean: run_osd td/ceph-helpers 1 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:56:34.690 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1433600f-1319-417a-8727-9855dc205477 2026-02-22T22:56:34.691 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1433600f-1319-417a-8727-9855dc205477' 2026-02-22T22:56:34.692 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 1433600f-1319-417a-8727-9855dc205477 2026-02-22T22:56:34.692 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:56:34.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiiZtp/vq3JhAA8p87sGxhMLRFZVWFXewDyQ== 2026-02-22T22:56:34.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiiZtp/vq3JhAA8p87sGxhMLRFZVWFXewDyQ=="}' 2026-02-22T22:56:34.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1433600f-1319-417a-8727-9855dc205477 -i td/ceph-helpers/1/new.json 2026-02-22T22:56:34.843 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:56:34.850 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:56:34.851 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiiZtp/vq3JhAA8p87sGxhMLRFZVWFXewDyQ== --osd-uuid 1433600f-1319-417a-8727-9855dc205477 2026-02-22T22:56:34.868 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:34.867+0000 7fde21825780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:34.870 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:34.868+0000 7fde21825780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:34.871 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:34.869+0000 7fde21825780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:34.871 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:34.869+0000 7fde21825780 -1 bdev(0x558cb0e65c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:56:34.871 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:34.869+0000 7fde21825780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:56:36.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:56:36.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:56:36.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:56:36.997 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:56:36.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:56:37.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:56:37.253 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:56:37.254 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:56:37.254 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:56:37.254 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:56:37.254 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:56:37.269 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:37.267+0000 7f906620d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:37.270 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:37.268+0000 7f906620d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:37.272 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:37.270+0000 7f906620d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:37.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:56:37.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:56:37.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:56:37.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:56:37.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:56:37.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:37.443 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:56:37.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:56:37.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:37.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:37.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:38.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:38.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:38.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:56:38.632 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:56:38.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:38.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:38.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:38.860 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:38.858+0000 7f906620d780 -1 Falling back to public interface 2026-02-22T22:56:39.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:39.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:39.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:56:39.823 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:56:39.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:39.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:40.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:41.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:41.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:41.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:56:41.014 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:56:41.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:41.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:41.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:41.803 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:41.801+0000 7f906620d780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:56:42.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:42.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:42.204 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:56:42.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:56:42.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:42.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:42.400 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:43.402 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:43.402 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:43.402 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:56:43.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:56:43.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:43.403 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/897067161,v1:127.0.0.1:6811/897067161] [v2:127.0.0.1:6812/897067161,v1:127.0.0.1:6813/897067161] exists,up 1433600f-1319-417a-8727-9855dc205477 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: test_wait_for_clean: wait_for_clean 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:56:43.591 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:56:43.592 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:56:43.592 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:56:43.592 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:56:43.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:56:43.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:56:43.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:56:43.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:56:43.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:56:43.676 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:56:43.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:56:43.863 INFO:tasks.workunit.client.0.trial123.stderr:1' 2026-02-22T22:56:43.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:56:43.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:56:43.863 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:56:43.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836485 2026-02-22T22:56:43.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836485 2026-02-22T22:56:43.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T22:56:43.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:56:43.927 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:56:43.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=68719476739 2026-02-22T22:56:43.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 68719476739 2026-02-22T22:56:43.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836485 1-68719476739' 2026-02-22T22:56:43.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:56:43.990 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836485 2026-02-22T22:56:43.990 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:56:43.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:56:43.991 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836485 2026-02-22T22:56:43.991 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:56:43.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836485 2026-02-22T22:56:43.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T22:56:43.993 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836485 2026-02-22T22:56:43.993 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:56:44.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-22T22:56:44.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:56:45.180 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:56:45.181 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:56:45.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-22T22:56:45.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:56:45.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-68719476739 2026-02-22T22:56:45.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:56:45.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:56:45.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-68719476739 2026-02-22T22:56:45.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:56:45.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=68719476739 2026-02-22T22:56:45.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 68719476739' 2026-02-22T22:56:45.375 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 68719476739 2026-02-22T22:56:45.375 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:56:45.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 68719476739 -lt 68719476739 2026-02-22T22:56:45.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:56:45.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:45.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:45.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:56:45.820 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:45.820 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:45.821 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:45.821 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:45.821 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:45.821 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:45.821 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:46.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:46.004 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:46.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:46.005 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:46.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:46.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' -1 2026-02-22T22:56:46.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: loop=0 2026-02-22T22:56:46.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: num_active_clean=0 2026-02-22T22:56:46.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:46.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:56:46.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:46.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:46.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:46.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:46.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:46.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:46.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:46.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:46.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:46.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:46.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:46.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:46.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:46.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:56:46.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:46.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:46.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:46.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:46.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:46.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:46.806 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:47.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:56:47.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:56:47.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 1 >= 13 )) 2026-02-22T22:56:47.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:47.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.2 2026-02-22T22:56:47.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:47.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:47.260 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:47.260 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:47.260 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:47.260 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:47.260 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:47.260 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:47.448 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:47.448 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:47.449 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:47.449 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:47.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:47.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:56:47.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:47.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:47.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:47.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:47.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:47.705 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:47.705 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:47.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:56:47.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:56:47.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 2 >= 13 )) 2026-02-22T22:56:47.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:47.963 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.4 2026-02-22T22:56:48.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:48.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:48.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:48.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:48.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:48.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:48.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:48.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:48.550 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:56:48.550 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:48.550 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:48.550 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:48.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:56:48.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:56:48.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:48.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:48.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:48.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:48.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:48.803 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:48.803 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:49.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=0 2026-02-22T22:56:49.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 0 '!=' null 2026-02-22T22:56:49.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: wait_for_clean: loop=0 2026-02-22T22:56:49.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:49.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:56:49.156 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:49.156 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:49.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:49.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:49.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:49.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:49.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:49.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:49.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=2 2026-02-22T22:56:49.347 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:49.347 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:49.347 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:49.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 2 = 4 2026-02-22T22:56:49.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 2 '!=' 0 2026-02-22T22:56:49.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: loop=0 2026-02-22T22:56:49.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: num_active_clean=2 2026-02-22T22:56:49.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:49.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:56:49.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:49.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:49.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:49.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:49.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:49.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:49.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:49.700 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:49.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=2 2026-02-22T22:56:49.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:49.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:49.889 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:50.138 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 2 = 4 2026-02-22T22:56:50.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 2 '!=' 2 2026-02-22T22:56:50.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:50.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:50.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:50.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:50.139 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:50.140 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:50.140 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:50.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=0 2026-02-22T22:56:50.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 0 '!=' null 2026-02-22T22:56:50.389 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: wait_for_clean: loop=0 2026-02-22T22:56:50.389 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:50.389 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:56:50.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:50.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:50.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:50.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:50.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:50.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:50.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:50.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:50.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=2 2026-02-22T22:56:50.676 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:50.676 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:50.676 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:50.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 2 = 4 2026-02-22T22:56:50.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 2 '!=' 2 2026-02-22T22:56:50.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:56:50.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:56:50.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:56:50.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:56:50.926 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:56:50.926 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:56:50.926 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:56:51.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2026-02-22T22:56:51.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2026-02-22T22:56:51.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: wait_for_clean: loop=0 2026-02-22T22:56:51.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:56:51.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:56:51.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:56:51.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:56:51.277 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:56:51.277 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:56:51.277 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:56:51.277 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:56:51.278 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:56:51.278 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:56:51.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:56:51.464 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:56:51.464 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:56:51.464 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:56:51.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:56:51.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:56:51.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:56:51.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: test_wait_for_clean: teardown td/ceph-helpers 2026-02-22T22:56:51.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:51.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:51.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:51.711 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:51.711 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:51.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:51.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:51.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:51.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:51.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:51.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:51.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:51.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:51.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:51.821 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:51.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:51.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:51.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:51.823 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:51.823 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:51.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:51.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:51.830 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:51.830 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:51.830 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:51.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_wait_for_health_ok td/ceph-helpers 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1863: test_wait_for_health_ok: local dir=td/ceph-helpers 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_wait_for_health_ok: setup td/ceph-helpers 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:56:51.833 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:56:51.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:56:51.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:56:51.834 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:56:51.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:56:51.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:56:51.834 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:56:51.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:56:51.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:56:51.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:56:51.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:56:51.836 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:56:51.836 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:56:51.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:56:51.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:51.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:56:51.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:56:51.837 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:56:51.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:56:51.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:56:51.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:56:51.840 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:56:51.841 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:51.841 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:51.841 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:56:51.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:56:51.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:56:51.842 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:56:51.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:56:51.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:51.843 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:51.843 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:56:51.844 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:56:51.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:56:51.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2026-02-22T22:56:51.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:56:51.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:51.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:51.864 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:51.865 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:51.865 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:51.865 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:51.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2026-02-22T22:56:51.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:56:51.891 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:51.892 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:51.893 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:56:51.938 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:56:51.939 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:56:51.940 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:51.940 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:51.940 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:56:51.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:56:51.940 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:56:51.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2026-02-22T22:56:51.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:56:51.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:56:51.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:56:51.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:56:51.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:56:51.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:56:52.086 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:56:52.086 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:52.086 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:52.086 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:52.086 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:52.086 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:52.086 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:52.087 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:56:52.087 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:56:52.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:56:52.108 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:56:52.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:56:52.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:56:52.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:56:52.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:56:52.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:56:52.110 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:56:52.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b101a883-ca4f-4972-83e4-f71d27a00d0d 2026-02-22T22:56:52.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b101a883-ca4f-4972-83e4-f71d27a00d0d' 2026-02-22T22:56:52.111 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 b101a883-ca4f-4972-83e4-f71d27a00d0d 2026-02-22T22:56:52.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:56:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0iZtpuX0IBxAA7ymNMJsQeCjGV22ZM2JKKw== 2026-02-22T22:56:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0iZtpuX0IBxAA7ymNMJsQeCjGV22ZM2JKKw=="}' 2026-02-22T22:56:52.120 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b101a883-ca4f-4972-83e4-f71d27a00d0d -i td/ceph-helpers/0/new.json 2026-02-22T22:56:52.222 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:56:52.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:56:52.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0iZtpuX0IBxAA7ymNMJsQeCjGV22ZM2JKKw== --osd-uuid b101a883-ca4f-4972-83e4-f71d27a00d0d 2026-02-22T22:56:52.245 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:52.244+0000 7f07fa709780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:52.247 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:52.245+0000 7f07fa709780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:52.248 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:52.246+0000 7f07fa709780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:52.248 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:52.246+0000 7f07fa709780 -1 bdev(0x562bdfb3bc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:56:52.248 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:52.246+0000 7f07fa709780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:56:55.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:56:55.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:56:55.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:56:55.135 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:56:55.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:56:55.387 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:56:55.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:56:55.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:56:55.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:56:55.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:56:55.388 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:56:55.404 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:55.402+0000 7f183b213780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:55.405 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:55.403+0000 7f183b213780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:55.406 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:55.404+0000 7f183b213780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:56:55.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:56:55.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:56:55.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:56:55.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:56:55.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:56:55.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:55.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:56:55.584 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:56:55.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:55.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:55.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:56.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:56.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:56.775 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:56:56.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:56:56.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:56.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:56.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:56.994 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:56.992+0000 7f183b213780 -1 Falling back to public interface 2026-02-22T22:56:57.966 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:57.966 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:57.967 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:56:57.967 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:56:57.967 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:57.967 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:58.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:59.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:56:59.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:56:59.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:56:59.161 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:56:59.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:56:59.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:56:59.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:56:59.690 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:56:59.688+0000 7f183b213780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:57:00.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:00.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:00.351 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:00.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:00.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:00.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:00.540 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/445620023,v1:127.0.0.1:6803/445620023] [v2:127.0.0.1:6804/445620023,v1:127.0.0.1:6805/445620023] exists,up b101a883-ca4f-4972-83e4-f71d27a00d0d 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:57:00.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b31a606-caf6-4ff4-81ea-8ca13cd60530 2026-02-22T22:57:00.546 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 1b31a606-caf6-4ff4-81ea-8ca13cd60530 2026-02-22T22:57:00.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1b31a606-caf6-4ff4-81ea-8ca13cd60530' 2026-02-22T22:57:00.547 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:57:00.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8iZtpyx8TIRAAAeVY4lb8xfyrZghdgugXjg== 2026-02-22T22:57:00.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8iZtpyx8TIRAAAeVY4lb8xfyrZghdgugXjg=="}' 2026-02-22T22:57:00.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b31a606-caf6-4ff4-81ea-8ca13cd60530 -i td/ceph-helpers/1/new.json 2026-02-22T22:57:00.750 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:00.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:57:00.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8iZtpyx8TIRAAAeVY4lb8xfyrZghdgugXjg== --osd-uuid 1b31a606-caf6-4ff4-81ea-8ca13cd60530 2026-02-22T22:57:00.775 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:00.772+0000 7f2c41216780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:00.776 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:00.773+0000 7f2c41216780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:00.777 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:00.774+0000 7f2c41216780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:00.777 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:00.775+0000 7f2c41216780 -1 bdev(0x565163321c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:57:00.778 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:00.775+0000 7f2c41216780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:57:02.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:57:02.905 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:57:02.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:57:02.906 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:57:02.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:57:03.161 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:57:03.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:57:03.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:57:03.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:57:03.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:57:03.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:57:03.178 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:03.175+0000 7f9739e0b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:03.179 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:03.176+0000 7f9739e0b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:03.180 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:03.177+0000 7f9739e0b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:03.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:57:03.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:03.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:57:03.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:03.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:03.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:03.350 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:03.350 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:03.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:03.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:03.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:04.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:04.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:04.537 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:04.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:57:04.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:04.537 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:04.726 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:04.764 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:04.761+0000 7f9739e0b780 -1 Falling back to public interface 2026-02-22T22:57:05.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:05.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:05.727 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:05.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:57:05.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:05.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:05.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:06.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:06.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:06.918 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:57:06.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:57:06.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:06.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:07.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:07.953 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:07.950+0000 7f9739e0b780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:57:08.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:08.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:08.108 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:08.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:08.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:08.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:08.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:09.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:09.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:09.304 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:57:09.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:57:09.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:09.304 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:09.495 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/599220439,v1:127.0.0.1:6811/599220439] [v2:127.0.0.1:6812/599220439,v1:127.0.0.1:6813/599220439] exists,up 1b31a606-caf6-4ff4-81ea-8ca13cd60530 2026-02-22T22:57:09.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:09.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:09.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-22T22:57:09.500 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:09.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b12bf07-f3f5-4429-b82b-94d717b3e76f 2026-02-22T22:57:09.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0b12bf07-f3f5-4429-b82b-94d717b3e76f' 2026-02-22T22:57:09.503 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 0b12bf07-f3f5-4429-b82b-94d717b3e76f 2026-02-22T22:57:09.503 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:57:09.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFiZtp3t5cHhAAOCC9JbxuoTWjXo/6dNq1Jw== 2026-02-22T22:57:09.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFiZtp3t5cHhAAOCC9JbxuoTWjXo/6dNq1Jw=="}' 2026-02-22T22:57:09.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b12bf07-f3f5-4429-b82b-94d717b3e76f -i td/ceph-helpers/2/new.json 2026-02-22T22:57:09.710 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:09.717 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2026-02-22T22:57:09.718 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFiZtp3t5cHhAAOCC9JbxuoTWjXo/6dNq1Jw== --osd-uuid 0b12bf07-f3f5-4429-b82b-94d717b3e76f 2026-02-22T22:57:09.737 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:09.735+0000 7f3aefe2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:09.738 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:09.736+0000 7f3aefe2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:09.739 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:09.737+0000 7f3aefe2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:09.739 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:09.738+0000 7f3aefe2c780 -1 bdev(0x5613ed4dfc00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:57:09.739 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:09.738+0000 7f3aefe2c780 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2026-02-22T22:57:11.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2026-02-22T22:57:11.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:57:11.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:57:11.858 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:57:11.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:57:12.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:57:12.117 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:57:12.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:57:12.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:57:12.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:57:12.118 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:57:12.134 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:12.132+0000 7f84e07ad780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:12.135 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:12.134+0000 7f84e07ad780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:12.136 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:12.135+0000 7f84e07ad780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:12.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:57:12.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:12.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:57:12.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:12.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:12.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:12.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:12.308 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:12.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:12.309 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:12.494 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:13.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:13.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:13.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:57:13.496 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:13.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:13.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:13.684 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:13.712 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:13.710+0000 7f84e07ad780 -1 Falling back to public interface 2026-02-22T22:57:14.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:14.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:14.686 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:14.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:57:14.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:14.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:14.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:15.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:15.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:15.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:57:15.879 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:57:15.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:15.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:16.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:16.412 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:16.410+0000 7f84e07ad780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:57:17.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:17.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:17.069 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:17.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:17.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:17.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:17.262 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3843926055,v1:127.0.0.1:6819/3843926055] [v2:127.0.0.1:6820/3843926055,v1:127.0.0.1:6821/3843926055] exists,up 0b12bf07-f3f5-4429-b82b-94d717b3e76f 2026-02-22T22:57:17.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:17.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:17.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:17.263 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2026-02-22T22:57:17.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:57:17.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:57:17.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:57:17.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:57:17.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:57:18.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:57:18.571 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_wait_for_health_ok: ceph osd down 0 2026-02-22T22:57:18.762 INFO:tasks.workunit.client.0.trial123.stderr:osd.0 is already down. 2026-02-22T22:57:18.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: test_wait_for_health_ok: TIMEOUT=1 2026-02-22T22:57:18.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: test_wait_for_health_ok: wait_for_health_ok 2026-02-22T22:57:18.770 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1859: wait_for_health_ok: wait_for_health HEALTH_OK 2026-02-22T22:57:18.770 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: wait_for_health: local grepstr=HEALTH_OK 2026-02-22T22:57:18.770 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: get_timeout_delays 1 .1 2026-02-22T22:57:18.771 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:57:18.771 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:57:18.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:57:18.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:57:18.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:57:18.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2026-02-22T22:57:18.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: local -a delays 2026-02-22T22:57:18.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: wait_for_health: local -i loop=0 2026-02-22T22:57:18.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: ceph health detail 2026-02-22T22:57:18.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: grep HEALTH_OK 2026-02-22T22:57:19.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1843: wait_for_health: (( 0 >= 4 )) 2026-02-22T22:57:19.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: wait_for_health: sleep 0.1 2026-02-22T22:57:19.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: wait_for_health: loop+=1 2026-02-22T22:57:19.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: ceph health detail 2026-02-22T22:57:19.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: grep HEALTH_OK 2026-02-22T22:57:19.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1843: wait_for_health: (( 1 >= 4 )) 2026-02-22T22:57:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: wait_for_health: sleep 0.2 2026-02-22T22:57:19.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: wait_for_health: loop+=1 2026-02-22T22:57:19.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: ceph health detail 2026-02-22T22:57:19.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: grep HEALTH_OK 2026-02-22T22:57:19.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1843: wait_for_health: (( 2 >= 4 )) 2026-02-22T22:57:19.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: wait_for_health: sleep 0.4 2026-02-22T22:57:20.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: wait_for_health: loop+=1 2026-02-22T22:57:20.264 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: ceph health detail 2026-02-22T22:57:20.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: grep HEALTH_OK 2026-02-22T22:57:20.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1843: wait_for_health: (( 3 >= 4 )) 2026-02-22T22:57:20.517 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: wait_for_health: sleep 0.3 2026-02-22T22:57:20.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: wait_for_health: loop+=1 2026-02-22T22:57:20.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: ceph health detail 2026-02-22T22:57:20.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: grep HEALTH_OK 2026-02-22T22:57:21.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1843: wait_for_health: (( 4 >= 4 )) 2026-02-22T22:57:21.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: wait_for_health: ceph health detail 2026-02-22T22:57:21.321 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2026-02-22T22:57:21.321 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] OSD_DOWN: 3 osds down 2026-02-22T22:57:21.321 INFO:tasks.workunit.client.0.trial123.stdout: osd.0 (root=default,host=trial123) is down 2026-02-22T22:57:21.321 INFO:tasks.workunit.client.0.trial123.stdout: osd.1 (root=default,host=trial123) is down 2026-02-22T22:57:21.322 INFO:tasks.workunit.client.0.trial123.stdout: osd.2 (root=default,host=trial123) is down 2026-02-22T22:57:21.322 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2026-02-22T22:57:21.322 INFO:tasks.workunit.client.0.trial123.stdout: host trial123 (root=default) (3 osds) is down 2026-02-22T22:57:21.322 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2026-02-22T22:57:21.322 INFO:tasks.workunit.client.0.trial123.stdout: root default (3 osds) is down 2026-02-22T22:57:21.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: wait_for_health: return 1 2026-02-22T22:57:21.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1859: wait_for_health_ok: return 1 2026-02-22T22:57:21.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2026-02-22T22:57:21.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2026-02-22T22:57:21.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:21.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:57:21.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:21.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:57:21.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:21.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:21.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:21.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:21.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:21.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:21.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:57:21.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:57:21.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:57:21.335 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:57:21.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T22:57:21.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:57:21.337 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-22T22:57:21.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T22:57:21.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:57:21.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:57:21.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:57:21.348 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:21.346+0000 7f304ae21780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:21.349 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:21.347+0000 7f304ae21780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:21.350 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:21.348+0000 7f304ae21780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:21.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T22:57:21.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:21.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:57:21.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:21.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:21.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:21.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:21.520 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:21.521 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:21.521 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:21.712 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:22.690 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:22.688+0000 7f304ae21780 -1 Falling back to public interface 2026-02-22T22:57:22.712 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:22.712 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:22.713 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:22.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:57:22.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:22.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:22.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:23.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:23.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:23.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:57:23.904 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:23.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:23.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:24.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:25.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:25.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:25.097 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:57:25.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:57:25.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:25.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:25.287 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:25.417 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:25.415+0000 7f304ae21780 -1 osd.0 13 log_to_monitors true 2026-02-22T22:57:26.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:26.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:26.289 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:26.289 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:26.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:26.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:26.477 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 16 up_thru 0 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/1489261267,v1:127.0.0.1:6803/1489261267] [v2:127.0.0.1:6804/1489261267,v1:127.0.0.1:6805/1489261267] exists,up b101a883-ca4f-4972-83e4-f71d27a00d0d 2026-02-22T22:57:26.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:26.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:26.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:26.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:57:26.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:26.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:57:26.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:57:26.485 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:57:26.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:57:26.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2026-02-22T22:57:26.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:57:26.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:57:26.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:57:26.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:57:26.498 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:26.496+0000 7f697c410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:26.499 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:26.497+0000 7f697c410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:26.500 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:26.498+0000 7f697c410780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:26.671 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:26.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:26.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:26.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:27.844 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:27.842+0000 7f697c410780 -1 Falling back to public interface 2026-02-22T22:57:27.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:27.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:27.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:57:27.863 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:27.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:27.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:28.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:29.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:29.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:29.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:57:29.053 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:29.053 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:29.053 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:29.242 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:30.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:30.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:30.244 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:57:30.244 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:57:30.244 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:30.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:30.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:30.550 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:30.548+0000 7f697c410780 -1 osd.1 11 log_to_monitors true 2026-02-22T22:57:31.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:31.433 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:31.434 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:31.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:31.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:31.434 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:31.623 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 14 last_clean_interval [9,11) [v2:127.0.0.1:6810/2282426963,v1:127.0.0.1:6811/2282426963] [v2:127.0.0.1:6812/2282426963,v1:127.0.0.1:6813/2282426963] exists,up 1b31a606-caf6-4ff4-81ea-8ca13cd60530 2026-02-22T22:57:31.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1879: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:31.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:31.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:31.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-22T22:57:31.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-22T22:57:31.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:31.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:57:31.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:31.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:57:31.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:31.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:31.629 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:31.629 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:31.629 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:31.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:31.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:31.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:31.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:31.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:31.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:31.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:31.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:31.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:31.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:57:31.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:57:31.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2026-02-22T22:57:31.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:57:31.632 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:57:31.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:57:31.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2026-02-22T22:57:31.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:57:31.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:57:31.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:57:31.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:57:31.641 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:31.639+0000 7f16884d9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:31.642 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:31.641+0000 7f16884d9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:31.643 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:31.642+0000 7f16884d9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:57:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:57:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:31.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:31.822 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:31.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:31.822 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:32.011 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:32.982 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:32.980+0000 7f16884d9780 -1 Falling back to public interface 2026-02-22T22:57:33.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:33.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:33.012 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:57:33.012 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:33.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:33.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:33.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:34.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:34.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:34.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:57:34.205 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:34.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:34.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:34.392 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:35.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:35.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:35.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:57:35.394 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:57:35.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:35.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:35.582 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:35.670 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:35.667+0000 7f16884d9780 -1 osd.2 13 log_to_monitors true 2026-02-22T22:57:36.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:36.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:36.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:36.583 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:36.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:36.584 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:57:36.773 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 14 last_clean_interval [0,0) [v2:127.0.0.1:6818/1364412235,v1:127.0.0.1:6819/1364412235] [v2:127.0.0.1:6820/1364412235,v1:127.0.0.1:6821/1364412235] exists,up 0b12bf07-f3f5-4429-b82b-94d717b3e76f 2026-02-22T22:57:36.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:36.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:36.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:36.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1880: test_wait_for_health_ok: wait_for_health_ok 2026-02-22T22:57:36.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1859: wait_for_health_ok: wait_for_health HEALTH_OK 2026-02-22T22:57:36.775 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: wait_for_health: local grepstr=HEALTH_OK 2026-02-22T22:57:36.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: get_timeout_delays 300 .1 2026-02-22T22:57:36.775 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:57:36.775 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:57:36.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:57:36.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:57:36.775 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:57:36.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2026-02-22T22:57:36.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: local -a delays 2026-02-22T22:57:36.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: wait_for_health: local -i loop=0 2026-02-22T22:57:36.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: ceph health detail 2026-02-22T22:57:36.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: wait_for_health: grep HEALTH_OK 2026-02-22T22:57:37.198 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_OK 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1881: test_wait_for_health_ok: teardown td/ceph-helpers 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:57:37.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:57:37.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:57:37.308 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:57:37.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:57:37.309 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:57:37.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:57:37.310 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:57:37.310 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:57:37.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:37.311 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:57:37.312 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:57:37.312 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:37.313 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:57:37.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:57:37.314 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:57:37.321 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:57:37.321 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.322 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:57:37.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:57:37.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:57:37.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:57:37.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_wait_for_osd td/ceph-helpers 2026-02-22T22:57:37.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2026-02-22T22:57:37.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2026-02-22T22:57:37.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:57:37.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:57:37.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:57:37.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:57:37.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:57:37.329 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:57:37.329 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:57:37.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:57:37.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:57:37.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:57:37.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:57:37.330 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:57:37.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:57:37.330 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:57:37.330 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:57:37.331 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:57:37.331 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:57:37.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:37.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:57:37.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:57:37.337 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:37.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:57:37.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:57:37.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:57:37.338 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:57:37.338 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.338 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:57:37.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:57:37.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:57:37.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:57:37.339 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:57:37.339 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.339 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.339 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:57:37.340 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:57:37.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:57:37.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:57:37.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:57:37.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:57:37.340 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:57:37.341 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:57:37.341 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:57:37.341 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:57:37.341 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:57:37.341 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:57:37.356 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:57:37.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:37.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:37.357 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:37.357 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.357 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:37.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:57:37.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:57:37.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:57:37.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:57:37.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:57:37.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:57:37.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:57:37.384 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:57:37.384 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:57:37.384 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:57:37.385 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:57:37.385 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.385 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.385 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:57:37.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:57:37.385 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.431 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.432 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:57:37.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:57:37.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:57:37.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2026-02-22T22:57:37.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:57:37.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:57:37.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:57:37.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:57:37.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:57:37.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:57:37.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:57:37.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:37.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:37.578 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:37.578 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.578 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.578 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:37.578 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:57:37.579 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:57:37.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2026-02-22T22:57:37.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:57:37.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:57:37.597 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:57:37.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:57:37.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:57:37.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:37.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:37.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:37.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:37.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:57:37.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:57:37.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:37.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:57:37.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:37.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:57:37.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:37.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:57:37.601 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:57:37.602 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=99fe980e-5698-4a8f-9a80-6a7815c1e90e 2026-02-22T22:57:37.602 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 99fe980e-5698-4a8f-9a80-6a7815c1e90e 2026-02-22T22:57:37.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 99fe980e-5698-4a8f-9a80-6a7815c1e90e' 2026-02-22T22:57:37.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:57:37.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhiZtpCN9BJBAAXYsXpCf+NuRYkfqMGDc6fg== 2026-02-22T22:57:37.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhiZtpCN9BJBAAXYsXpCf+NuRYkfqMGDc6fg=="}' 2026-02-22T22:57:37.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 99fe980e-5698-4a8f-9a80-6a7815c1e90e -i td/ceph-helpers/0/new.json 2026-02-22T22:57:37.712 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:37.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:57:37.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhiZtpCN9BJBAAXYsXpCf+NuRYkfqMGDc6fg== --osd-uuid 99fe980e-5698-4a8f-9a80-6a7815c1e90e 2026-02-22T22:57:37.737 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:37.734+0000 7f33ba553780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:37.739 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:37.736+0000 7f33ba553780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:37.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:37.737+0000 7f33ba553780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:37.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:37.737+0000 7f33ba553780 -1 bdev(0x563b49385c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:57:37.740 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:37.737+0000 7f33ba553780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:57:40.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:57:40.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:57:40.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:57:40.397 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:57:40.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:57:40.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:57:40.655 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:57:40.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:57:40.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:57:40.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:57:40.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:57:40.672 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:40.669+0000 7f09a6f95780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:40.673 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:40.670+0000 7f09a6f95780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:40.674 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:40.671+0000 7f09a6f95780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:40.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:57:40.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:40.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:57:40.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:40.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:40.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:40.845 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:40.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:40.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:40.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:41.033 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:42.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:42.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:42.036 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:42.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:57:42.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:42.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:42.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:42.255 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:42.252+0000 7f09a6f95780 -1 Falling back to public interface 2026-02-22T22:57:43.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:43.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:43.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:57:43.231 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:43.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:43.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:43.422 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:44.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:44.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:44.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:57:44.423 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:57:44.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:44.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:44.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:45.197 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:45.195+0000 7f09a6f95780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:57:45.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:45.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:45.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:45.613 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:45.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:45.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:45.807 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/11537233,v1:127.0.0.1:6803/11537233] [v2:127.0.0.1:6804/11537233,v1:127.0.0.1:6805/11537233] exists,up 99fe980e-5698-4a8f-9a80-6a7815c1e90e 2026-02-22T22:57:45.811 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:45.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:45.813 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:45.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:57:45.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:57:45.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:57:45.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:57:45.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=854f37f0-d297-40e2-819a-e8c9ef30b2a4 2026-02-22T22:57:45.815 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 854f37f0-d297-40e2-819a-e8c9ef30b2a4' 2026-02-22T22:57:45.815 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 854f37f0-d297-40e2-819a-e8c9ef30b2a4 2026-02-22T22:57:45.815 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:57:45.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpiZtpSHn/MBAAB2HPQVru3E9XpPuWkQRHtQ== 2026-02-22T22:57:45.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpiZtpSHn/MBAAB2HPQVru3E9XpPuWkQRHtQ=="}' 2026-02-22T22:57:45.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 854f37f0-d297-40e2-819a-e8c9ef30b2a4 -i td/ceph-helpers/1/new.json 2026-02-22T22:57:46.020 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:46.026 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:57:46.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpiZtpSHn/MBAAB2HPQVru3E9XpPuWkQRHtQ== --osd-uuid 854f37f0-d297-40e2-819a-e8c9ef30b2a4 2026-02-22T22:57:46.045 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:46.043+0000 7f39bfd3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:46.046 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:46.044+0000 7f39bfd3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:46.047 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:46.045+0000 7f39bfd3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:46.047 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:46.046+0000 7f39bfd3f780 -1 bdev(0x55f9d2f05c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:57:46.048 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:46.046+0000 7f39bfd3f780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:57:48.160 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:57:48.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:57:48.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:57:48.161 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:57:48.162 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:57:48.413 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:57:48.414 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:57:48.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:57:48.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:57:48.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:57:48.416 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:57:48.430 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:48.428+0000 7fecdba11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:48.431 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:48.429+0000 7fecdba11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:48.432 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:48.430+0000 7fecdba11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:48.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:57:48.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:48.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:57:48.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:48.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:48.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:48.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:48.604 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:48.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:48.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:48.789 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:49.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:49.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:49.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:57:49.791 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:57:49.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:49.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:49.979 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:50.018 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:50.016+0000 7fecdba11780 -1 Falling back to public interface 2026-02-22T22:57:50.980 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:50.980 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:50.981 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:57:50.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:57:50.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:50.982 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:51.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:52.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:52.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:52.172 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:57:52.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:57:52.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:52.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:52.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:52.966 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:52.964+0000 7fecdba11780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:57:53.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:53.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:53.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:57:53.361 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:57:53.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:53.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:53.552 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:54.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:54.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:54.554 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:57:54.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:57:54.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:54.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3195363434,v1:127.0.0.1:6811/3195363434] [v2:127.0.0.1:6812/3195363434,v1:127.0.0.1:6813/3195363434] exists,up 854f37f0-d297-40e2-819a-e8c9ef30b2a4 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:54.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:54.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:54.744 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:54.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:54.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:54.744 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:54.931 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/11537233,v1:127.0.0.1:6803/11537233] [v2:127.0.0.1:6804/11537233,v1:127.0.0.1:6805/11537233] exists,up 99fe980e-5698-4a8f-9a80-6a7815c1e90e 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:54.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:54.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:54.934 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:54.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:54.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:57:55.120 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3195363434,v1:127.0.0.1:6811/3195363434] [v2:127.0.0.1:6812/3195363434,v1:127.0.0.1:6813/3195363434] exists,up 854f37f0-d297-40e2-819a-e8c9ef30b2a4 2026-02-22T22:57:55.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:55.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:55.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:55.121 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2026-02-22T22:57:55.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:57:55.121 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:57:55.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:57:55.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:57:55.122 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:57:56.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:57:56.430 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2026-02-22T22:57:56.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2026-02-22T22:57:56.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:57:56.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:56.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:56.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:57:56.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:56.432 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:56.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:56.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2026-02-22T22:57:56.624 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 11 last_clean_interval [0,0) [v2:127.0.0.1:6802/11537233,v1:127.0.0.1:6803/11537233] [v2:127.0.0.1:6804/11537233,v1:127.0.0.1:6805/11537233] exists 99fe980e-5698-4a8f-9a80-6a7815c1e90e 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:57:56.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:57:56.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:57:56.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2026-02-22T22:57:56.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:57:56.627 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:56.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:57:56.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:57:56.816 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:57:57.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:57:57.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:57:57.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:57:57.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:57:57.819 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:57:57.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:57:57.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:57:57.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:57:57.929 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:57:57.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:57:57.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:57:57.931 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:57:57.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:57.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:57:57.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:57:57.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:57.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:57:57.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:57:57.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:57:57.939 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:57:57.939 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:57.939 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:57.939 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_wait_for_peered td/ceph-helpers 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1775: test_wait_for_peered: local dir=td/ceph-helpers 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1777: test_wait_for_peered: setup td/ceph-helpers 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:57:57.940 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:57:57.941 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:57:57.941 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:57:57.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:57:57.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:57:57.941 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:57:57.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:57:57.943 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:57:57.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:57:57.944 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:57:57.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:57:57.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:57:57.945 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:57:57.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:57.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:57:57.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:57:57.946 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:57:57.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:57:57.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:57:57.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:57:57.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:57:57.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:57.949 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:57.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:57:57.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:57:57.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:57:57.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:57:57.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:57:57.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:57.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:57.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:57:57.953 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1778: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:57:57.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-22T22:57:57.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:57:57.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:57.973 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:57.974 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:57.974 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:57.974 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:57.974 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:57.974 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T22:57:57.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:57:58.000 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:57:58.000 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:57:58.000 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:57:58.000 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:57:58.000 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:57:58.000 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:57:58.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:57:58.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:57:58.001 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:57:58.001 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:58.001 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:58.001 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:57:58.001 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:57:58.002 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:57:58.046 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:57:58.047 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:57:58.048 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:58.048 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:58.048 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:57:58.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:57:58.048 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:57:58.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: test_wait_for_peered: run_osd td/ceph-helpers 0 2026-02-22T22:57:58.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:57:58.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:57:58.098 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:57:58.099 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:57:58.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9bf4278-410d-482b-a6d3-221877586104 2026-02-22T22:57:58.100 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c9bf4278-410d-482b-a6d3-221877586104' 2026-02-22T22:57:58.100 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 c9bf4278-410d-482b-a6d3-221877586104 2026-02-22T22:57:58.101 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:57:58.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2iZtp825qBhAAkhOTLR7mYtVLUkwtQKxzXw== 2026-02-22T22:57:58.109 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2iZtp825qBhAAkhOTLR7mYtVLUkwtQKxzXw=="}' 2026-02-22T22:57:58.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9bf4278-410d-482b-a6d3-221877586104 -i td/ceph-helpers/0/new.json 2026-02-22T22:57:58.209 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:57:58.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:57:58.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2iZtp825qBhAAkhOTLR7mYtVLUkwtQKxzXw== --osd-uuid c9bf4278-410d-482b-a6d3-221877586104 2026-02-22T22:57:58.231 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:58.229+0000 7fa1420d5780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:58.232 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:58.230+0000 7fa1420d5780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:58.233 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:58.231+0000 7fa1420d5780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:57:58.233 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:58.232+0000 7fa1420d5780 -1 bdev(0x55ebafb89c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:57:58.234 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:57:58.232+0000 7fa1420d5780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:58:00.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:58:00.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:58:00.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:58:00.361 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:58:00.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:58:00.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:58:00.461 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:58:00.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:58:00.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:58:00.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:58:00.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:58:00.475 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:00.474+0000 7fb53e282780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:00.477 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:00.475+0000 7fb53e282780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:00.478 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:00.476+0000 7fb53e282780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:00.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:58:00.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:58:00.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:58:00.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:58:00.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:58:00.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:00.560 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:00.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:58:00.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:00.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:00.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:01.556 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:01.554+0000 7fb53e282780 -1 Falling back to public interface 2026-02-22T22:58:01.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:01.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:01.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:58:01.658 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:58:01.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:01.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:01.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:02.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:02.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:02.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:58:02.758 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:58:02.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:02.758 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:02.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:03.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:03.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:03.858 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:58:03.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:58:03.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:03.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:03.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:04.272 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:04.270+0000 7fb53e282780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:58:04.956 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:04.956 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:04.957 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:58:04.957 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:58:04.958 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:04.958 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:05.057 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2776297150,v1:127.0.0.1:6801/2776297150] [v2:127.0.0.1:6802/2776297150,v1:127.0.0.1:6803/2776297150] exists,up c9bf4278-410d-482b-a6d3-221877586104 2026-02-22T22:58:05.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:58:05.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:58:05.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:58:05.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1780: test_wait_for_peered: run_mgr td/ceph-helpers x 2026-02-22T22:58:05.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:58:05.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:58:05.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:58:05.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:58:05.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:58:05.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:58:05.156 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:58:05.156 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:05.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:05.157 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:05.157 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:05.157 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:05.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:05.157 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:05.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:05.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1781: test_wait_for_peered: create_rbd_pool 2026-02-22T22:58:05.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:58:05.270 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:58:05.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:58:05.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:58:05.453 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:58:05.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:58:06.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: test_wait_for_peered: wait_for_clean 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 1 .1 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:58:06.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:58:06.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2026-02-22T22:58:06.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:58:06.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:58:06.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:58:06.764 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:58:06.764 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:58:06.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:58:06.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:58:06.866 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:58:06.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:58:06.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=0 2026-02-22T22:58:06.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 0 2026-02-22T22:58:06.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-0' 2026-02-22T22:58:06.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:58:06.929 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-0 2026-02-22T22:58:06.929 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:58:06.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:58:06.930 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-0 2026-02-22T22:58:06.930 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:58:06.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=0 2026-02-22T22:58:06.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 0' 2026-02-22T22:58:06.932 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 0 2026-02-22T22:58:06.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:58:07.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 0 -lt 0 2026-02-22T22:58:07.030 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:58:07.031 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:07.031 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:07.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 == 0 2026-02-22T22:58:07.132 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: sleep 1 2026-02-22T22:58:08.134 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:58:08.135 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:08.135 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:58:08.386 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:08.572 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:58:08.572 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:58:08.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:08.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:08.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:58:08.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' -1 2026-02-22T22:58:08.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: loop=0 2026-02-22T22:58:08.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: num_active_clean=0 2026-02-22T22:58:08.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:58:08.826 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:58:08.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:58:08.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:58:08.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:58:08.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:58:08.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:58:08.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:58:08.929 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:58:08.929 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:09.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:58:09.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:58:09.111 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:09.112 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:09.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:58:09.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:58:09.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:58:09.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:58:09.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:58:09.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:58:09.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:58:09.366 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:58:09.366 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:58:09.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:58:09.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:58:09.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 1 >= 4 )) 2026-02-22T22:58:09.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:58:09.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.2 2026-02-22T22:58:09.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:58:09.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:58:09.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:58:09.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:58:09.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:58:09.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:58:09.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:58:09.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:10.008 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:58:10.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:58:10.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:10.009 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:58:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:58:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:58:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:58:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:58:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:58:10.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:58:10.263 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:58:10.263 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:58:10.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:58:10.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:58:10.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 2 >= 4 )) 2026-02-22T22:58:10.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:58:10.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.4 2026-02-22T22:58:10.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:58:10.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:58:10.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:58:10.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:58:10.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:58:10.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:58:10.922 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:58:10.922 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:11.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:58:11.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:58:11.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:11.107 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:11.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:58:11.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:58:11.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:58:11.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:58:11.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:58:11.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:58:11.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:58:11.362 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:58:11.362 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:58:11.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:58:11.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:58:11.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 3 >= 4 )) 2026-02-22T22:58:11.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:58:11.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.3 2026-02-22T22:58:11.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:58:11.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:58:11.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:58:11.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:58:11.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:58:11.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:58:11.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:58:11.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:12.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=0 2026-02-22T22:58:12.105 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:58:12.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:12.106 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:12.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 0 = 4 2026-02-22T22:58:12.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 0 '!=' 0 2026-02-22T22:58:12.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:58:12.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:58:12.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:58:12.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:58:12.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:58:12.359 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:58:12.359 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:58:12.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:58:12.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:58:12.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 4 >= 4 )) 2026-02-22T22:58:12.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: wait_for_clean: ceph report 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout:{ 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_fingerprint": "b647181d-b1aa-4e52-b564-a1d9ad3cc385", 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "version": "19.2.3-689-gf90f4eae", 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "commit": "f90f4eaebb28b619cf76e80cbbe997656da69579", 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "timestamp": "2026-02-22T22:58:12.858743+0000", 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "tag": "", 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "health": { 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "status": "HEALTH_WARN", 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "checks": { 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "PG_DEGRADED": { 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:58:12.860 INFO:tasks.workunit.client.0.trial123.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "count": 2 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [ 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "TOO_FEW_OSDS": { 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "severity": "HEALTH_WARN", 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "summary": { 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "count": 2 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "detail": [], 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "muted": false 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "mutes": [] 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "monmap_first_committed": 1, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "monmap_last_committed": 1, 2026-02-22T22:58:12.861 INFO:tasks.workunit.client.0.trial123.stdout: "monmap": { 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "fsid": "d116af49-5d19-4d5a-91be-c8a20698ede8", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:57:57.966670Z", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "created": "2026-02-22T22:57:57.966670Z", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "min_mon_release": 19, 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "min_mon_release_name": "squid", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "election_strategy": 1, 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "disallowed_leaders": "", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode": false, 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "tiebreaker_mon": "", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "removed_ranks": "", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "features": { 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "persistent": [ 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "kraken", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "luminous", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "mimic", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap-prune", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "nautilus", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "octopus", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "pacific", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "elector-pinging", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "quincy", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "reef", 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "squid" 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "optional": [] 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.862 INFO:tasks.workunit.client.0.trial123.stdout: "mons": [ 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "rank": 0, 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "name": "a", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "public_addrs": { 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:7109", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 0 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:7109/0", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "public_addr": "127.0.0.1:7109/0", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "priority": 0, 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 0, 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "crush_location": "{}" 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "quorum": [ 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: 0 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap": { 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 11, 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "fsid": "d116af49-5d19-4d5a-91be-c8a20698ede8", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "created": "2026-02-22T22:57:58.000428+0000", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:58:06.732429+0000", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "last_up_change": "2026-02-22T22:58:04.865255+0000", 2026-02-22T22:58:12.863 INFO:tasks.workunit.client.0.trial123.stdout: "last_in_change": "2026-02-22T22:57:58.186142+0000", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "flags_num": 5799936, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "flags_set": [ 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "pglog_hardlimit", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snapdirs", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "recovery_deletes", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "sortbitwise" 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "crush_version": 3, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "full_ratio": 0.99000000953674316, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "backfillfull_ratio": 0.99000000953674316, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "nearfull_ratio": 0.85000002384185791, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_snapshot": "", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "pool_max": 1, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "max_osd": 1, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "require_min_compat_client": "luminous", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "min_compat_client": "jewel", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "require_osd_release": "squid", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "allow_crimson": false, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "pools": [ 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "pool_name": "rbd", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "create_time": "2026-02-22T22:58:05.356332+0000", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "flags": 8193, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "type": 1, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "size": 2, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "min_size": 1, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "crush_rule": 0, 2026-02-22T22:58:12.864 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_count": 0, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_target": 0, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_barrier": 0, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "is_stretch_pool": false, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "object_hash": 2, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "pg_autoscale_mode": "off", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num": 4, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "pg_placement_num": 4, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "pg_placement_num_target": 4, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num_target": 4, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "pg_num_pending": 4, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "last_pg_merge_meta": { 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "source_pgid": "0.0", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "ready_epoch": 0, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_started": 0, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_clean": 0, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "source_version": "0'0", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "target_version": "0'0" 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "last_change": "11", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend": "0", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend_prenautilus": "0", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "last_force_op_resend_preluminous": "0", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "auid": 0, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "snap_mode": "selfmanaged", 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "snap_seq": 2, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "snap_epoch": 11, 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "pool_snaps": [], 2026-02-22T22:58:12.865 INFO:tasks.workunit.client.0.trial123.stdout: "removed_snaps": "[]", 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "quota_max_bytes": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "quota_max_objects": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "tiers": [], 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "tier_of": -1, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "read_tier": -1, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "write_tier": -1, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "cache_mode": "none", 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "target_max_bytes": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "target_max_objects": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_dirty_ratio_micro": 400000, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "cache_target_full_ratio_micro": 800000, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "cache_min_flush_age": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "cache_min_evict_age": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "erasure_code_profile": "", 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_params": { 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "type": "none" 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_period": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_count": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "use_gmt_hitset": true, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "min_read_recency_for_promote": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "min_write_recency_for_promote": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_grade_decay_rate": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "hit_set_search_last_n": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "grade_table": [], 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "stripe_width": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "expected_num_objects": 0, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "fast_read": false, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "options": {}, 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "application_metadata": { 2026-02-22T22:58:12.866 INFO:tasks.workunit.client.0.trial123.stdout: "rbd": {} 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "read_balance": { 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "score_type": "Fair distribution", 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "score_acting": 1, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "score_stable": 1, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "optimal_score": 1, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "raw_score_acting": 1, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "raw_score_stable": 1, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "primary_affinity_weighted": 1, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "average_primary_affinity": 1, 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "average_primary_affinity_weighted": 1 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "osds": [ 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.867 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "uuid": "c9bf4278-410d-482b-a6d3-221877586104", 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "up": 1, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "in": 1, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 1, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "primary_affinity": 1, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "last_clean_begin": 0, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "last_clean_end": 0, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 5, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "up_thru": 6, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "down_at": 0, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "lost_at": 0, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "public_addrs": { 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6800", 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6801", 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.868 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_addrs": { 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6802", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6803", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_back_addrs": { 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6806", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6807", 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_front_addrs": { 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: "addrvec": [ 2026-02-22T22:58:12.869 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v2", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6804", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "type": "v1", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "addr": "127.0.0.1:6805", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "nonce": 2776297150 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "public_addr": "127.0.0.1:6801/2776297150", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "cluster_addr": "127.0.0.1:6803/2776297150", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_back_addr": "127.0.0.1:6807/2776297150", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "heartbeat_front_addr": "127.0.0.1:6805/2776297150", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "state": [ 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "exists", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "up" 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "osd_xinfo": [ 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "down_stamp": "0.000000", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "laggy_probability": 0, 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "laggy_interval": 0, 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "features": 4540701547738038271, 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "old_weight": 0, 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "last_purged_snaps_scrub": "0.000000", 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "dead_epoch": 0 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap": [], 2026-02-22T22:58:12.870 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap_items": [], 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "pg_upmap_primaries": [], 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "pg_temp": [], 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "primary_temp": [], 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "blocklist": {}, 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "range_blocklist": {}, 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "erasure_code_profiles": { 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "default": { 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "k": "2", 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "m": "2", 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "plugin": "jerasure", 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "technique": "reed_sol_van" 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "removed_snaps_queue": [ 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "snaps": [ 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "begin": 2, 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "length": 1 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "new_removed_snaps": [ 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "snaps": [ 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "begin": 2, 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "length": 1 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.871 INFO:tasks.workunit.client.0.trial123.stdout: "new_purged_snaps": [], 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "crush_node_flags": {}, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "device_class_flags": {}, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode": { 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode_enabled": false, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_bucket_count": 0, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "degraded_stretch_mode": 0, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "recovering_stretch_mode": 0, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "stretch_mode_bucket": 0 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "osd_metadata": [ 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "arch": "x86_64", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "back_addr": "[v2:127.0.0.1:6802/2776297150,v1:127.0.0.1:6803/2776297150]", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "back_iface": "", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs": "1", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_dedicated_db": "0", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_dedicated_wal": "0", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluefs_single_shared_device": "1", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_allocation_from_file": "1", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_allocator": "hybrid", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_access_mode": "file", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_block_size": "4096", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_devices": "nvme1n1", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_driver": "KernelDevice", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_optimal_io_size": "4096", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_rotational": "0", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_size": "107374182400", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_support_discard": "1", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_bdev_type": "ssd", 2026-02-22T22:58:12.872 INFO:tasks.workunit.client.0.trial123.stdout: "bluestore_min_alloc_size": "4096", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_release": "squid", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version": "ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable)", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version_short": "19.2.3-689-gf90f4eae", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "ceph_version_when_created": "ceph version 19.2.3-689-gf90f4eae (f90f4eaebb28b619cf76e80cbbe997656da69579) squid (stable)", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "cpu": "AMD EPYC 4464P 12-Core Processor", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "created_at": "2026-02-22T22:57:59.831377Z", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "default_device_class": "ssd", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "device_ids": "nvme1n1=Micron_7450_MTFDKBA960TFR_25174FCD7341", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "device_paths": "nvme1n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "devices": "nvme1n1", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "distro": "centos", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "distro_description": "CentOS Stream 9", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "distro_version": "9", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "front_addr": "[v2:127.0.0.1:6800/2776297150,v1:127.0.0.1:6801/2776297150]", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "front_iface": "", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/2776297150,v1:127.0.0.1:6807/2776297150]", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/2776297150,v1:127.0.0.1:6805/2776297150]", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "hostname": "trial123", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "journal_rotational": "0", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Feb 11 20:19:22 UTC 2026", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "kernel_version": "5.14.0-681.el9.x86_64", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "mem_swap_kb": "8388604", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "mem_total_kb": "130438940", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "objectstore_numa_node": "0", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "objectstore_numa_nodes": "0", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "os": "Linux", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "osd_data": "td/ceph-helpers/0", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "osd_objectstore": "bluestore", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "osdspec_affinity": "", 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "rotational": "0" 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_clean_epochs": { 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "min_last_epoch_clean": 0, 2026-02-22T22:58:12.873 INFO:tasks.workunit.client.0.trial123.stdout: "last_epoch_clean": { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "per_pool": [] 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "osd_epochs": [ 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 5 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_first_committed": 1, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "osdmap_last_committed": 11, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "crushmap": { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "devices": [ 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "name": "osd.0", 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "class": "ssd" 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "types": [ 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 0, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "name": "osd" 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "name": "host" 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 2, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "name": "chassis" 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 3, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "name": "rack" 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.874 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 4, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "row" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 5, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "pdu" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 6, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "pod" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 7, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "room" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 8, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "datacenter" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 9, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "zone" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 10, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "region" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "root" 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "buckets": [ 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "id": -1, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "name": "default", 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "root", 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.875 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "id": -3, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "id": -2, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "name": "default~ssd", 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 11, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "root", 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "id": -4, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "id": -3, 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "name": "trial123", 2026-02-22T22:58:12.876 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "host", 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "id": -4, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "name": "trial123~ssd", 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "type_id": 1, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "type_name": "host", 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "alg": "straw2", 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "hash": "rjenkins1", 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "items": [ 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "id": 0, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "weight": 6402, 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: "pos": 0 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.877 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "rules": [ 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "rule_id": 0, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "rule_name": "replicated_rule", 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "type": 1, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "steps": [ 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "op": "take", 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "item": -1, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "item_name": "default" 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "op": "choose_firstn", 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "num": 0, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "type": "osd" 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "op": "emit" 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: ] 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "tunables": { 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "choose_local_tries": 0, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "choose_local_fallback_tries": 0, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "choose_total_tries": 50, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_descend_once": 1, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_vary_r": 1, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "chooseleaf_stable": 1, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "msr_descents": 100, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "msr_collision_tries": 100, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "straw_calc_version": 1, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "allowed_bucket_algs": 54, 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "profile": "jewel", 2026-02-22T22:58:12.878 INFO:tasks.workunit.client.0.trial123.stdout: "optimal_tunables": 1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "legacy_tunables": 0, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "minimum_required_version": "jewel", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables": 1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables2": 1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "has_v2_rules": 0, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables3": 1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "has_v3_rules": 0, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "has_v4_buckets": 1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "require_feature_tunables5": 1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "has_v5_rules": 0, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "has_msr_rules": 0 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "choose_args": {} 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "fsmap": { 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "btime": "2026-02-22T22:57:58:000263+0000", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "default_fscid": -1, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "compat": { 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "compat": {}, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "ro_compat": {}, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "incompat": { 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_1": "base v0.20", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_2": "client writeable ranges", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_3": "default file layouts on dirs", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_4": "dir inode in separate object", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_5": "mds uses versioned encoding", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_6": "dirfrag is stored in omap", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_8": "no anchor table", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_9": "file layout v2", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_10": "snaprealm v2", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_11": "minor log segments", 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_12": "quiesce subvolumes" 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "feature_flags": { 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "enable_multiple": true, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "ever_enabled_multiple": true 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.879 INFO:tasks.workunit.client.0.trial123.stdout: "standbys": [], 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "filesystems": [] 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "mdsmap_first_committed": 1, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "mdsmap_last_committed": 1, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "auth": { 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "first_committed": 1, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "last_committed": 3, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_secrets": 1 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg": 4, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_active": 4, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_unknown": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_osd": 1, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "pool_sum": { 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "stat_sum": { 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes": 19, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects": 2, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_clones": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_copies": 4, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_degraded": 2, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_misplaced": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_unfound": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_dirty": 2, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_whiteouts": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_read": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_read_kb": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_write": 2, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_write_kb": 2, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_scrub_errors": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_recovered": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_recovered": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_keys_recovered": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_omap": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:58:12.880 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_kb": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_kb": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_promote": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_high": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_low": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_some": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_full": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_pinned": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_large_omap_objects": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_manifest": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_bytes": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_keys": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_repaired": 0 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "store_stats": { 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "total": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "available": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 0 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "log_size": 3, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "ondisk_log_size": 3, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "up": 4, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "acting": 4, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_store_stats": 0 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "osd_sum": { 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 0, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_pgs": 4, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_osds": 1, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_osds": 1, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "kb": 104857600, 2026-02-22T22:58:12.881 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used": 27048, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_data": 72, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_omap": 1, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_meta": 26942, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "kb_avail": 104830552, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "statfs": { 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "total": 107374182400, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "available": 107346485248, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 73728, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 14676, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 1593, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 27589063 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "hb_peers": [], 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "snap_trim_queue_len": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "num_snap_trimming": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "num_shards_repaired": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "op_queue_age_hist": { 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "histogram": [], 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "upper_bound": 1 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "perf_stat": { 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ms": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ms": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ns": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ns": 0 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "alerts": [], 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "network_ping_times": [] 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "osd_sum_by_class": { 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "ssd": { 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "up_from": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 0, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "num_pgs": 4, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "num_osds": 1, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_osds": 1, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T22:58:12.882 INFO:tasks.workunit.client.0.trial123.stdout: "kb": 104857600, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used": 27048, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_data": 72, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_omap": 1, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "kb_used_meta": 26942, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "kb_avail": 104830552, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "statfs": { 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "total": 107374182400, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "available": 107346485248, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 73728, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 14676, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 1593, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 27589063 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "hb_peers": [], 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "snap_trim_queue_len": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "num_snap_trimming": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "num_shards_repaired": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "op_queue_age_hist": { 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "histogram": [], 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "upper_bound": 1 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "perf_stat": { 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ms": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ms": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "commit_latency_ns": 0, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "apply_latency_ns": 0 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "alerts": [], 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "network_ping_times": [] 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "pool_stats": [ 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "poolid": 1, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg": 4, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "stat_sum": { 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes": 19, 2026-02-22T22:58:12.883 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects": 2, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_clones": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_object_copies": 4, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_missing": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_degraded": 2, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_misplaced": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_unfound": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_dirty": 2, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_whiteouts": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_read": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_read_kb": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_write": 2, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_write_kb": 2, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_scrub_errors": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_recovered": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_recovered": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_keys_recovered": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_omap": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_kb": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_kb": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_promote": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_high": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_flush_mode_low": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_some": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_evict_mode_full": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_pinned": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_large_omap_objects": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_manifest": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_bytes": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_omap_keys": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "num_objects_repaired": 0 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "store_stats": { 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "total": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "available": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "internally_reserved": 0, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "allocated": 4096, 2026-02-22T22:58:12.884 INFO:tasks.workunit.client.0.trial123.stdout: "data_stored": 19, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed": 0, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_allocated": 0, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "data_compressed_original": 0, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "omap_allocated": 0, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "internal_metadata": 0 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "log_size": 3, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "ondisk_log_size": 3, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "up": 4, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "acting": 4, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num_store_stats": 1 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "osd_stats": [ 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "seq": 21474836481 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_by_state": [ 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "state": "active+undersized+degraded", 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num": 2 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "state": "active+undersized", 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num": 2 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num_pg_by_osd": [ 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "osd": 0, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num_primary_pg": 4, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num_acting_pg": 4, 2026-02-22T22:58:12.885 INFO:tasks.workunit.client.0.trial123.stdout: "num_up_not_acting_pg": 0 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snaps": [ 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: { 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "pool": 1, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "purged_snaps": {} 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: ], 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "servicemap": { 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "epoch": 1, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "modified": "2026-02-22T22:57:58.000596+0000", 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "services": {} 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: }, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "mgrstat_first_committed": 1, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "mgrstat_last_committed": 4, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "logm_first_committed": 1, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "logm_last_committed": 56, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "paxos": { 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "first_committed": 1, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "last_committed": 80, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "last_pn": 0, 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: "accepted_pn": 0 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout: } 2026-02-22T22:58:12.886 INFO:tasks.workunit.client.0.trial123.stdout:} 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:report 3920532947 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: wait_for_clean: return 1 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: test_wait_for_peered: run_osd td/ceph-helpers 1 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:58:12.887 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=955c69d4-06cb-4af6-bdf1-5e372e2f7e2c 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 955c69d4-06cb-4af6-bdf1-5e372e2f7e2c 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 955c69d4-06cb-4af6-bdf1-5e372e2f7e2c' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEiptpYU2FNBAAYFMnWKH50UJKZpuF5fydXA== 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEiptpYU2FNBAAYFMnWKH50UJKZpuF5fydXA=="}' 2026-02-22T22:58:12.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 955c69d4-06cb-4af6-bdf1-5e372e2f7e2c -i td/ceph-helpers/1/new.json 2026-02-22T22:58:13.072 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:58:13.080 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2026-02-22T22:58:13.081 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEiptpYU2FNBAAYFMnWKH50UJKZpuF5fydXA== --osd-uuid 955c69d4-06cb-4af6-bdf1-5e372e2f7e2c 2026-02-22T22:58:13.098 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:13.096+0000 7fbf9f310780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:13.100 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:13.097+0000 7fbf9f310780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:13.101 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:13.098+0000 7fbf9f310780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:13.101 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:13.098+0000 7fbf9f310780 -1 bdev(0x563732fffc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:58:13.101 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:13.098+0000 7fbf9f310780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-22T22:58:15.256 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-22T22:58:15.256 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:58:15.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:58:15.258 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:58:15.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:58:15.515 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:58:15.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:58:15.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:58:15.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:58:15.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:58:15.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:58:15.532 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:15.529+0000 7f5558433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:15.533 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:15.530+0000 7f5558433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:15.534 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:15.531+0000 7f5558433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:15.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:58:15.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:58:15.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:58:15.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:58:15.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:58:15.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:15.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:58:15.703 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:15.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:15.704 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:58:15.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:16.891 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:16.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:16.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:58:16.892 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:58:16.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:16.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:58:17.082 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:17.115 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:17.112+0000 7f5558433780 -1 Falling back to public interface 2026-02-22T22:58:18.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:18.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:18.084 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:58:18.084 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:58:18.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:18.085 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:58:18.274 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:19.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:19.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:19.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:58:19.275 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:58:19.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:19.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:58:19.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:20.065 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:20.064+0000 7f5558433780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:58:20.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:20.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:20.463 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:58:20.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:58:20.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:20.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:58:20.654 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:21.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:21.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:21.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:58:21.656 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:58:21.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:21.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:58:21.845 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 15 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1391949735,v1:127.0.0.1:6811/1391949735] [v2:127.0.0.1:6812/1391949735,v1:127.0.0.1:6813/1391949735] exists,up 955c69d4-06cb-4af6-bdf1-5e372e2f7e2c 2026-02-22T22:58:21.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:58:21.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:58:21.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:58:21.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1784: test_wait_for_peered: wait_for_peered 2026-02-22T22:58:21.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: wait_for_peered: local cmd= 2026-02-22T22:58:21.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: wait_for_peered: local num_peered=-1 2026-02-22T22:58:21.849 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: wait_for_peered: local cur_peered 2026-02-22T22:58:21.849 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: wait_for_peered: get_timeout_delays 90 .1 2026-02-22T22:58:21.850 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:58:21.850 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:58:21.850 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:58:21.850 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:58:21.850 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:58:21.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:58:21.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: wait_for_peered: local -a delays 2026-02-22T22:58:21.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: wait_for_peered: local -i loop=0 2026-02-22T22:58:21.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: flush_pg_stats 2026-02-22T22:58:21.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:58:21.931 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:58:22.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:58:22.123 INFO:tasks.workunit.client.0.trial123.stderr:1' 2026-02-22T22:58:22.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:58:22.123 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:58:22.123 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:58:22.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836485 2026-02-22T22:58:22.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836485 2026-02-22T22:58:22.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T22:58:22.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:58:22.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:58:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509444 2026-02-22T22:58:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509444 2026-02-22T22:58:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836485 1-64424509444' 2026-02-22T22:58:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:58:22.252 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836485 2026-02-22T22:58:22.252 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:58:22.253 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:58:22.254 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836485 2026-02-22T22:58:22.254 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:58:22.255 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836485 2026-02-22T22:58:22.255 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T22:58:22.255 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836485 2026-02-22T22:58:22.256 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:58:22.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-22T22:58:22.442 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:58:23.443 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:58:23.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:58:23.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-22T22:58:23.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:58:23.634 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-64424509444 2026-02-22T22:58:23.635 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:58:23.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:58:23.636 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-64424509444 2026-02-22T22:58:23.636 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:58:23.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509444 2026-02-22T22:58:23.637 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 64424509444 2026-02-22T22:58:23.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 64424509444' 2026-02-22T22:58:23.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:58:23.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509444 -lt 64424509444 2026-02-22T22:58:23.826 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: get_num_pgs 2026-02-22T22:58:23.826 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:23.826 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:24.078 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: test 4 == 0 2026-02-22T22:58:24.078 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1751: wait_for_peered: true 2026-02-22T22:58:24.078 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1755: wait_for_peered: get_num_active_or_peered 2026-02-22T22:58:24.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2026-02-22T22:58:24.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-22T22:58:24.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-22T22:58:24.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-22T22:58:24.079 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:24.262 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1755: wait_for_peered: cur_peered=4 2026-02-22T22:58:24.262 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1756: wait_for_peered: get_num_pgs 2026-02-22T22:58:24.263 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:24.263 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1756: wait_for_peered: test 4 = 4 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1756: wait_for_peered: break 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1771: wait_for_peered: return 0 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: test_wait_for_peered: teardown td/ceph-helpers 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:24.510 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:24.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:24.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:24.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:24.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:24.617 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:58:24.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:58:24.618 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:58:24.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:58:24.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:58:24.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:58:24.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:24.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:58:24.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:58:24.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:24.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:58:24.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:58:24.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:58:24.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:58:24.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_wait_for_scrub td/ceph-helpers 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: test_wait_for_scrub: local dir=td/ceph-helpers 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: test_wait_for_scrub: setup td/ceph-helpers 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:58:24.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:58:24.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:24.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:24.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:24.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:24.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:24.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:24.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:58:24.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:58:24.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:58:24.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:58:24.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:58:24.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:58:24.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:24.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:58:24.634 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:58:24.635 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:24.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:58:24.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:58:24.636 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:58:24.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:58:24.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.638 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:58:24.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:58:24.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:58:24.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:58:24.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:58:24.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:58:24.641 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:58:24.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:58:24.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:58:24.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:24.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:24.662 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:24.662 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.662 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:24.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:58:24.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:58:24.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:58:24.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:58:24.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:58:24.689 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.690 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:58:24.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:58:24.691 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:58:24.737 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:58:24.738 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.738 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.738 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:58:24.738 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:58:24.738 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:58:24.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: test_wait_for_scrub: run_mgr td/ceph-helpers x 2026-02-22T22:58:24.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:58:24.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:58:24.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:58:24.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:58:24.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:58:24.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:58:24.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:58:24.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:24.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:24.880 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:24.880 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.881 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.881 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:24.881 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:24.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:24.900 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2091: test_wait_for_scrub: run_osd td/ceph-helpers 0 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:58:24.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:24.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:58:24.903 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cdb89d03-1302-4dfb-ae21-0623bfd2e009 2026-02-22T22:58:24.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cdb89d03-1302-4dfb-ae21-0623bfd2e009' 2026-02-22T22:58:24.904 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 cdb89d03-1302-4dfb-ae21-0623bfd2e009 2026-02-22T22:58:24.904 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:58:24.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQiptpQ7k+NhAA+NMjy+8M18xfEoxj3Kjw6Q== 2026-02-22T22:58:24.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQiptpQ7k+NhAA+NMjy+8M18xfEoxj3Kjw6Q=="}' 2026-02-22T22:58:24.912 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cdb89d03-1302-4dfb-ae21-0623bfd2e009 -i td/ceph-helpers/0/new.json 2026-02-22T22:58:25.016 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:25.022 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:58:25.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQiptpQ7k+NhAA+NMjy+8M18xfEoxj3Kjw6Q== --osd-uuid cdb89d03-1302-4dfb-ae21-0623bfd2e009 2026-02-22T22:58:25.041 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:25.039+0000 7ff88f831780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:25.042 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:25.041+0000 7ff88f831780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:25.043 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:25.041+0000 7ff88f831780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:25.044 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:25.042+0000 7ff88f831780 -1 bdev(0x5608866f1c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:58:25.044 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:25.042+0000 7ff88f831780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:58:27.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:58:27.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:58:27.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:58:27.934 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:58:27.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:58:28.190 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:58:28.191 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:58:28.191 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:58:28.191 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:58:28.191 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:58:28.191 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:58:28.206 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:28.204+0000 7f28dbc20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:28.208 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:28.206+0000 7f28dbc20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:28.209 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:28.207+0000 7f28dbc20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:28.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:58:28.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:58:28.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:58:28.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:58:28.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:58:28.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:28.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:58:28.377 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:28.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:28.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:28.568 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:29.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:29.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:29.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:58:29.569 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:58:29.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:29.569 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:29.760 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:29.804 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:29.802+0000 7f28dbc20780 -1 Falling back to public interface 2026-02-22T22:58:30.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:30.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:30.761 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:58:30.762 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:58:30.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:30.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:30.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:31.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:31.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:31.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:58:31.954 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:58:31.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:31.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:32.144 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:32.747 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:32.745+0000 7f28dbc20780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:58:33.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:33.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:33.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:58:33.147 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:58:33.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:33.147 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:33.334 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:34.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:34.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:34.336 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:58:34.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:58:34.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:34.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:34.524 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2489724351,v1:127.0.0.1:6803/2489724351] [v2:127.0.0.1:6804/2489724351,v1:127.0.0.1:6805/2489724351] exists,up cdb89d03-1302-4dfb-ae21-0623bfd2e009 2026-02-22T22:58:34.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:58:34.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:58:34.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:58:34.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: test_wait_for_scrub: create_rbd_pool 2026-02-22T22:58:34.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:58:34.712 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:58:34.719 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:58:34.720 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:58:34.957 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:58:34.964 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:58:35.965 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:58:36.248 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2093: test_wait_for_scrub: wait_for_clean 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:58:36.249 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:58:36.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:58:36.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:58:36.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:58:36.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:58:36.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:58:36.332 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:58:36.522 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:58:36.522 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:58:36.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:58:36.523 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:58:36.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836484 2026-02-22T22:58:36.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836484 2026-02-22T22:58:36.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T22:58:36.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:58:36.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836484 2026-02-22T22:58:36.586 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:58:36.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:58:36.588 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836484 2026-02-22T22:58:36.588 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:58:36.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836484 2026-02-22T22:58:36.589 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836484 2026-02-22T22:58:36.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T22:58:36.590 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:58:36.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:58:36.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:58:37.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:58:37.784 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:58:37.975 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T22:58:37.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:58:37.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:37.976 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:38.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:58:38.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:58:38.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:58:38.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:58:38.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:58:38.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:58:38.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:58:38.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:38.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:58:38.431 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:58:38.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:38.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:38.681 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:58:38.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:58:38.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:58:38.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: test_wait_for_scrub: local pgid=1.0 2026-02-22T22:58:38.682 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: test_wait_for_scrub: ceph pg repair 1.0 2026-02-22T22:58:38.862 INFO:tasks.workunit.client.0.trial123.stderr:instructing pg 1.0 on osd.0 to repair 2026-02-22T22:58:38.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: test_wait_for_scrub: get_last_scrub_stamp 1.0 2026-02-22T22:58:38.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:58:38.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:58:38.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:58:38.870 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:58:39.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: test_wait_for_scrub: local last_scrub=2026-02-22T22:58:34.906334+0000 2026-02-22T22:58:39.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: test_wait_for_scrub: wait_for_scrub 1.0 2026-02-22T22:58:34.906334+0000 2026-02-22T22:58:39.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: wait_for_scrub: local pgid=1.0 2026-02-22T22:58:39.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: local last_scrub=2026-02-22T22:58:34.906334+0000 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i=0 )) 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:58:39.062 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:58:39.247 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:58:34.906334+0000 '>' 2026-02-22T22:58:34.906334+0000 2026-02-22T22:58:39.248 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:58:40.248 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:58:40.248 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 300 )) 2026-02-22T22:58:40.250 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:58:40.250 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:58:40.250 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:58:40.251 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:58:40.251 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:58:38.839951+0000 '>' 2026-02-22T22:58:34.906334+0000 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: wait_for_scrub: return 0 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:40.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:40.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:40.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: test_wait_for_scrub: get_last_scrub_stamp 1.0 2026-02-22T22:58:40.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:58:40.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:58:40.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:58:40.546 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: test_wait_for_scrub: last_scrub=2026-02-22T22:58:38.839951+0000 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: test_wait_for_scrub: TIMEOUT=1 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: test_wait_for_scrub: wait_for_scrub 1.0 2026-02-22T22:58:38.839951+0000 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: wait_for_scrub: local pgid=1.0 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: local last_scrub=2026-02-22T22:58:38.839951+0000 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i=0 )) 2026-02-22T22:58:40.731 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 1 )) 2026-02-22T22:58:40.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T22:58:40.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2026-02-22T22:58:40.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T22:58:40.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T22:58:40.732 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T22:58:40.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: wait_for_scrub: test 2026-02-22T22:58:38.839951+0000 '>' 2026-02-22T22:58:38.839951+0000 2026-02-22T22:58:40.913 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: wait_for_scrub: sleep 1 2026-02-22T22:58:41.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i++ )) 2026-02-22T22:58:41.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: wait_for_scrub: (( i < 1 )) 2026-02-22T22:58:41.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: wait_for_scrub: return 1 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2101: test_wait_for_scrub: teardown td/ceph-helpers 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:41.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:42.023 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:42.023 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:58:42.024 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:58:42.024 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:58:42.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:58:42.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:58:42.026 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:58:42.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:42.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:58:42.027 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:58:42.028 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:42.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:58:42.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:58:42.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:58:42.033 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:58:42.033 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.033 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.034 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:58:42.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:58:42.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: run_tests: for func in $funcs 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:42.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:42.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:42.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:42.039 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:58:42.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:58:42.040 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:58:42.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:58:42.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:58:42.040 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:58:42.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:42.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:58:42.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:58:42.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:42.041 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:58:42.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:58:42.042 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:58:42.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:58:42.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.043 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.043 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:58:42.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:58:42.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:58:42.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2026-02-22T22:58:42.045 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:58:42.045 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.045 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83985 2026-02-22T22:58:42.046 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:58:42.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:58:42.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2026-02-22T22:58:42.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:58:42.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:58:42.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:42.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:42.069 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:42.070 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.070 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:42.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T22:58:42.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:58:42.096 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:58:42.096 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:58:42.096 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:58:42.096 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.097 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:58:42.098 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:58:42.098 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get fsid 2026-02-22T22:58:42.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:58:42.142 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.143 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83985/ceph-mon.a.asok 2026-02-22T22:58:42.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:58:42.144 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83985/ceph-mon.a.asok config get mon_host 2026-02-22T22:58:42.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2026-02-22T22:58:42.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2026-02-22T22:58:42.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:58:42.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:58:42.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:58:42.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2026-02-22T22:58:42.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:58:42.286 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:58:42.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:42.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:42.287 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:42.287 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.287 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.287 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:42.288 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:42.288 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:58:42.306 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:58:42.307 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:58:42.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:58:42.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:58:42.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2026-02-22T22:58:42.308 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:58:42.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=32d0d133-5d58-4f81-bc6e-dcd6dc518f7e 2026-02-22T22:58:42.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 32d0d133-5d58-4f81-bc6e-dcd6dc518f7e' 2026-02-22T22:58:42.308 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 32d0d133-5d58-4f81-bc6e-dcd6dc518f7e 2026-02-22T22:58:42.309 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:58:42.320 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiiptpnSwDExAANjkNezv8ASz4O9rqubBaCw== 2026-02-22T22:58:42.320 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiiptpnSwDExAANjkNezv8ASz4O9rqubBaCw=="}' 2026-02-22T22:58:42.320 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 32d0d133-5d58-4f81-bc6e-dcd6dc518f7e -i td/ceph-helpers/0/new.json 2026-02-22T22:58:42.421 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:42.428 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2026-02-22T22:58:42.429 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiiptpnSwDExAANjkNezv8ASz4O9rqubBaCw== --osd-uuid 32d0d133-5d58-4f81-bc6e-dcd6dc518f7e 2026-02-22T22:58:42.446 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:42.444+0000 7f8698413780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:42.447 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:42.445+0000 7f8698413780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:42.448 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:42.446+0000 7f8698413780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:42.448 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:42.447+0000 7f8698413780 -1 bdev(0x55a7478cdc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:58:42.449 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:42.447+0000 7f8698413780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-22T22:58:45.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-22T22:58:45.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:58:45.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:58:45.354 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:58:45.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:58:45.611 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:58:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:58:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=d116af49-5d19-4d5a-91be-c8a20698ede8 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.83985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:58:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:58:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:58:45.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:58:45.628 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:45.625+0000 7fd729c23780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:45.629 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:45.626+0000 7fd729c23780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:45.630 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:45.627+0000 7fd729c23780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:45.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:58:45.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:58:45.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:58:45.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:58:45.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:58:45.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:45.800 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:45.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:58:45.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:45.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:45.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:46.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:46.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:46.990 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:58:46.990 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:58:46.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:46.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:47.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:47.227 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:47.224+0000 7fd729c23780 -1 Falling back to public interface 2026-02-22T22:58:48.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:48.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:48.178 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:58:48.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:58:48.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:48.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:48.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:49.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:49.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:49.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:58:49.368 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:58:49.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:49.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:49.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:50.419 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:50.416+0000 7fd729c23780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:58:50.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:50.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:50.559 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:58:50.559 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:58:50.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:50.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:50.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:58:51.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:58:51.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:51.752 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:58:51.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:58:51.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:51.752 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:51.942 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3678171202,v1:127.0.0.1:6803/3678171202] [v2:127.0.0.1:6804/3678171202,v1:127.0.0.1:6805/3678171202] exists,up 32d0d133-5d58-4f81-bc6e-dcd6dc518f7e 2026-02-22T22:58:51.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:58:51.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:58:51.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:58:51.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2026-02-22T22:58:51.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T22:58:52.127 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' does not exist 2026-02-22T22:58:52.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T22:58:52.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:58:52.372 INFO:tasks.workunit.client.0.trial123.stderr:pool 'rbd' already exists 2026-02-22T22:58:52.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:58:53.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:58:53.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:58:53.663 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:58:53.663 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:58:53.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:58:53.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:58:53.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:58:53.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:58:53.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:58:53.748 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:58:53.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids=0 2026-02-22T22:58:53.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:58:53.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:58:53.934 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:58:53.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836484 2026-02-22T22:58:53.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836484 2026-02-22T22:58:53.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T22:58:53.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:58:53.997 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836484 2026-02-22T22:58:53.997 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:58:53.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:58:53.998 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836484 2026-02-22T22:58:53.999 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:58:54.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836484 2026-02-22T22:58:54.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T22:58:54.000 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836484 2026-02-22T22:58:54.000 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:58:54.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T22:58:54.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:58:55.187 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:58:55.187 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:58:55.375 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T22:58:55.376 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:58:55.376 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:55.376 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:55.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T22:58:55.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:58:55.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:58:55.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:58:55.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:58:55.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:58:55.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:58:55.633 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:58:55.818 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:58:55.818 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:58:55.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:58:55.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2026-02-22T22:58:56.068 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2026-02-22T22:58:56.069 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:56.069 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2026-02-22T22:58:56.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2026-02-22T22:58:56.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2026-02-22T22:58:56.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2026-02-22T22:58:56.069 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2026-02-22T22:58:56.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2026-02-22T22:58:56.256 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2026-02-22T22:58:56.256 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2026-02-22T22:58:56.256 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2026-02-22T22:58:56.257 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2026-02-22T22:58:56.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:58:56.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2026-02-22T22:58:56.258 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:56.258 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:56.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:56.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:56.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:56.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:56.363 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:58:56.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:58:56.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:58:56.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:58:56.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:58:56.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:58:56.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:56.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:58:56.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:58:56.367 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:56.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:58:56.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:58:56.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2026-02-22T22:58:56.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:58:56.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83985 2026-02-22T22:58:56.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83985 2026-02-22T22:58:56.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:58:56.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:58:56.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: : exit 0 2026-02-22T22:58:56.375 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T22:58:56.375 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T22:58:56.442 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2026-02-22T22:58:56.443 DEBUG:teuthology.orchestra.run.trial123:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2026-02-22T22:58:56.504 INFO:tasks.workunit.client.0.trial123.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T22:58:56.507 INFO:tasks.workunit.client.0.trial123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T22:58:56.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:58:56.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T22:58:56.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/test-snaptrim-stats 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2026-02-22T22:58:56.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2026-02-22T22:58:56.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none ' 2026-02-22T22:58:56.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:TEST_snaptrim_stats_multiple_snaps' 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:58:56.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T22:58:56.512 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:56.512 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:56.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:56.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:56.512 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:56.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:56.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:58:56.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:58:56.513 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:58:56.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:58:56.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:58:56.514 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:58:56.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:56.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:58:56.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:58:56.515 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:56.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:58:56.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:58:56.516 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T22:58:56.518 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:58:56.518 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.518 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T22:58:56.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:58:56.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:58:56.519 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2026-02-22T22:58:56.520 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:58:56.520 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.520 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.175810 2026-02-22T22:58:56.521 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:58:56.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T22:58:56.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T22:58:56.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:58:56.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-22T22:58:56.523 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.524 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:58:56.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T22:58:56.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:58:56.525 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:58:56.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:58:56.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:58:56.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:58:56.525 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:58:56.526 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:58:56.526 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:58:56.526 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:58:56.526 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:58:56.526 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:58:56.526 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:58:56.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:56.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:58:56.527 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:58:56.527 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:58:56.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:58:56.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:58:56.528 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T22:58:56.529 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:58:56.529 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.530 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T22:58:56.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:58:56.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:58:56.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2026-02-22T22:58:56.531 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:58:56.531 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.532 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.532 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.175810 2026-02-22T22:58:56.532 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2026-02-22T22:58:56.534 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:58:56.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T22:58:56.577 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:58:56.578 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:58:56.578 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:58:56.578 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:58:56.579 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:58:56.579 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:58:56.580 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:58:56.580 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:58:56.580 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:58:56.580 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:58:56.580 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.580 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.580 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175810/ceph-mon.a.asok 2026-02-22T22:58:56.581 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:58:56.581 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.175810/ceph-mon.a.asok config get fsid 2026-02-22T22:58:56.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:58:56.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:58:56.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:58:56.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:58:56.624 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175810/ceph-mon.a.asok 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:58:56.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.175810/ceph-mon.a.asok config get mon_host 2026-02-22T22:58:56.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2026-02-22T22:58:56.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:58:56.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:58:56.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:58:56.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2026-02-22T22:58:56.672 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:58:56.770 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:58:56.770 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:56.770 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:56.771 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:56.771 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.771 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:58:56.771 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:56.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:58:56.789 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2026-02-22T22:58:56.790 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:58:56.791 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2026-02-22T22:58:56.792 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:58:56.793 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3566436-691d-4f3f-a6c5-3cd473c91b8d 2026-02-22T22:58:56.793 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b3566436-691d-4f3f-a6c5-3cd473c91b8d' 2026-02-22T22:58:56.793 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 b3566436-691d-4f3f-a6c5-3cd473c91b8d 2026-02-22T22:58:56.793 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:58:56.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwiptpiFzbLxAA7MvTWB+3kJMNnF1egLSGRw== 2026-02-22T22:58:56.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwiptpiFzbLxAA7MvTWB+3kJMNnF1egLSGRw=="}' 2026-02-22T22:58:56.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3566436-691d-4f3f-a6c5-3cd473c91b8d -i td/test-snaptrim-stats/0/new.json 2026-02-22T22:58:56.909 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:56.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2026-02-22T22:58:56.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAwiptpiFzbLxAA7MvTWB+3kJMNnF1egLSGRw== --osd-uuid b3566436-691d-4f3f-a6c5-3cd473c91b8d 2026-02-22T22:58:56.933 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:56.932+0000 7f7b11cb4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:56.935 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:56.933+0000 7f7b11cb4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:56.936 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:56.934+0000 7f7b11cb4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:56.936 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:56.934+0000 7f7b11cb4780 -1 bdev(0x563031915c00 td/test-snaptrim-stats/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:58:56.936 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:56.935+0000 7f7b11cb4780 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2026-02-22T22:58:59.345 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2026-02-22T22:58:59.346 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:58:59.346 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:58:59.347 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:58:59.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:58:59.604 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:58:59.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:58:59.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:58:59.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:58:59.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:58:59.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:58:59.621 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:59.619+0000 7f9197a1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:59.622 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:59.620+0000 7f9197a1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:59.623 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:58:59.621+0000 7f9197a1f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:58:59.793 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:58:59.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:58:59.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:58:59.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:58:59.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:58:59.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:58:59.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:58:59.795 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:58:59.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:58:59.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:58:59.983 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:00.984 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:00.984 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:00.984 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:00.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:59:00.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:00.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:01.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:01.215 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:01.213+0000 7f9197a1f780 -1 Falling back to public interface 2026-02-22T22:59:02.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:02.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:02.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:59:02.175 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:59:02.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:02.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:02.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:03.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:03.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:03.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:59:03.364 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:59:03.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:03.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:03.552 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:04.171 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:04.169+0000 7f9197a1f780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:59:04.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:04.553 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:04.553 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:59:04.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:59:04.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:04.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:04.754 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:05.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:05.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:05.756 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:59:05.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:59:05.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:05.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:05.945 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/328248975,v1:127.0.0.1:6803/328248975] [v2:127.0.0.1:6804/328248975,v1:127.0.0.1:6805/328248975] exists,up b3566436-691d-4f3f-a6c5-3cd473c91b8d 2026-02-22T22:59:05.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:59:05.949 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:59:05.950 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:05.951 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:59:05.952 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:59:05.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:05.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2026-02-22T22:59:05.953 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:59:05.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fb9df8a-2e7e-4351-aa66-672303f7cc7e 2026-02-22T22:59:05.953 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3fb9df8a-2e7e-4351-aa66-672303f7cc7e' 2026-02-22T22:59:05.953 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 3fb9df8a-2e7e-4351-aa66-672303f7cc7e 2026-02-22T22:59:05.953 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:59:05.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5iptpOvYjORAADET22yK0OP7jGR1KGGbakg== 2026-02-22T22:59:05.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5iptpOvYjORAADET22yK0OP7jGR1KGGbakg=="}' 2026-02-22T22:59:05.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fb9df8a-2e7e-4351-aa66-672303f7cc7e -i td/test-snaptrim-stats/1/new.json 2026-02-22T22:59:06.150 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:06.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2026-02-22T22:59:06.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQA5iptpOvYjORAADET22yK0OP7jGR1KGGbakg== --osd-uuid 3fb9df8a-2e7e-4351-aa66-672303f7cc7e 2026-02-22T22:59:06.175 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:06.173+0000 7f99576b4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:06.176 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:06.175+0000 7f99576b4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:06.177 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:06.175+0000 7f99576b4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:06.178 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:06.176+0000 7f99576b4780 -1 bdev(0x56347b931c00 td/test-snaptrim-stats/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:59:06.178 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:06.176+0000 7f99576b4780 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2026-02-22T22:59:08.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2026-02-22T22:59:08.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:59:08.294 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:59:08.294 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:59:08.295 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:59:08.546 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:59:08.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:59:08.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:08.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:59:08.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:59:08.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:59:08.562 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:08.561+0000 7fcf4a83f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:08.564 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:08.562+0000 7fcf4a83f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:08.565 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:08.563+0000 7fcf4a83f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:08.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:59:08.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:59:08.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:59:08.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:59:08.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:59:08.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:08.734 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:59:08.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:59:08.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:08.735 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:08.919 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:09.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:09.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:09.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:59:09.920 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:09.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:09.921 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:10.105 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:10.154 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:10.152+0000 7fcf4a83f780 -1 Falling back to public interface 2026-02-22T22:59:11.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:11.106 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:11.106 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:59:11.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:59:11.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:11.107 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:11.295 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:12.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:12.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:12.296 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:59:12.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:59:12.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:12.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:12.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:12.856 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:12.854+0000 7fcf4a83f780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:59:13.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:13.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:13.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:59:13.483 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:59:13.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:13.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:13.670 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3898097804,v1:127.0.0.1:6811/3898097804] [v2:127.0.0.1:6812/3898097804,v1:127.0.0.1:6813/3898097804] exists,up 3fb9df8a-2e7e-4351-aa66-672303f7cc7e 2026-02-22T22:59:13.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:59:13.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:59:13.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:59:13.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T22:59:13.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:13.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2026-02-22T22:59:13.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:13.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:13.678 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2026-02-22T22:59:13.679 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:59:13.679 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e4d9dc4-53e4-4951-a769-4b716914727f 2026-02-22T22:59:13.679 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 3e4d9dc4-53e4-4951-a769-4b716914727f 2026-02-22T22:59:13.680 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3e4d9dc4-53e4-4951-a769-4b716914727f' 2026-02-22T22:59:13.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:59:13.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBiptpbCnAKBAA0HRSrjKrCBUYK4LMKGXRlQ== 2026-02-22T22:59:13.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBiptpbCnAKBAA0HRSrjKrCBUYK4LMKGXRlQ=="}' 2026-02-22T22:59:13.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e4d9dc4-53e4-4951-a769-4b716914727f -i td/test-snaptrim-stats/2/new.json 2026-02-22T22:59:13.874 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:59:13.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2026-02-22T22:59:13.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBBiptpbCnAKBAA0HRSrjKrCBUYK4LMKGXRlQ== --osd-uuid 3e4d9dc4-53e4-4951-a769-4b716914727f 2026-02-22T22:59:13.899 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:13.897+0000 7f0ca16e4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:13.900 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:13.898+0000 7f0ca16e4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:13.901 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:13.899+0000 7f0ca16e4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:13.902 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:13.900+0000 7f0ca16e4780 -1 bdev(0x5611b3a7dc00 td/test-snaptrim-stats/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:59:13.902 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:13.900+0000 7f0ca16e4780 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2026-02-22T22:59:16.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2026-02-22T22:59:16.029 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:59:16.030 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:59:16.030 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T22:59:16.031 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:59:16.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:59:16.283 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T22:59:16.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:16.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:59:16.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:59:16.284 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:59:16.299 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:16.297+0000 7f943e60c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:16.300 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:16.298+0000 7f943e60c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:16.301 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:16.299+0000 7f943e60c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:16.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:59:16.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:59:16.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:59:16.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:59:16.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:59:16.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:16.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:59:16.472 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:59:16.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:16.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:59:16.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:17.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:17.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:17.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:59:17.658 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:17.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:17.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:59:17.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:17.903 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:17.901+0000 7f943e60c780 -1 Falling back to public interface 2026-02-22T22:59:18.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:18.846 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:18.847 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:59:18.847 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:59:18.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:18.848 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:59:19.034 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:20.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:20.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:20.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:59:20.036 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:59:20.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:20.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:59:20.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:20.593 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:20.590+0000 7f943e60c780 -1 osd.2 0 log_to_monitors true 2026-02-22T22:59:21.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:21.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:21.223 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:59:21.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:59:21.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:21.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:59:21.408 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1030773521,v1:127.0.0.1:6819/1030773521] [v2:127.0.0.1:6820/1030773521,v1:127.0.0.1:6821/1030773521] exists,up 3e4d9dc4-53e4-4951-a769-4b716914727f 2026-02-22T22:59:21.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:59:21.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:59:21.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:59:21.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2026-02-22T22:59:21.706 INFO:tasks.workunit.client.0.trial123.stderr:noscrub is set 2026-02-22T22:59:21.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2026-02-22T22:59:22.010 INFO:tasks.workunit.client.0.trial123.stderr:nodeep-scrub is set 2026-02-22T22:59:22.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2026-02-22T22:59:22.019 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2026-02-22T22:59:22.261 INFO:tasks.workunit.client.0.trial123.stderr:pool 'test' already exists 2026-02-22T22:59:22.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:59:23.270 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2026-02-22T22:59:23.270 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:59:23.270 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:59:23.270 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:59:23.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:59:23.271 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:59:23.271 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:59:23.271 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:59:23.272 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:59:23.272 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:59:23.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:59:23.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:59:23.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:59:23.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:59:23.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:59:23.354 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:59:23.539 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:59:23.539 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:59:23.539 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T22:59:23.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:59:23.540 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:59:23.540 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:59:23.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836487 2026-02-22T22:59:23.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836487 2026-02-22T22:59:23.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T22:59:23.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:59:23.601 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:59:23.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=38654705668 2026-02-22T22:59:23.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 38654705668 2026-02-22T22:59:23.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836487 1-38654705668' 2026-02-22T22:59:23.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:59:23.664 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:59:23.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=55834574851 2026-02-22T22:59:23.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 55834574851 2026-02-22T22:59:23.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836487 1-38654705668 2-55834574851' 2026-02-22T22:59:23.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:59:23.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836487 2026-02-22T22:59:23.728 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:59:23.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:59:23.729 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836487 2026-02-22T22:59:23.729 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:59:23.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836487 2026-02-22T22:59:23.730 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T22:59:23.730 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836487 2026-02-22T22:59:23.730 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:59:23.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836485 -lt 21474836487 2026-02-22T22:59:23.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:59:24.918 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:59:24.919 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:59:25.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-22T22:59:25.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:59:25.109 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-38654705668 2026-02-22T22:59:25.109 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:59:25.110 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:59:25.110 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-38654705668 2026-02-22T22:59:25.110 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:59:25.111 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=38654705668 2026-02-22T22:59:25.112 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-22T22:59:25.112 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 38654705668 2026-02-22T22:59:25.112 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:59:25.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 38654705669 -lt 38654705668 2026-02-22T22:59:25.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:59:25.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-55834574851 2026-02-22T22:59:25.300 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:59:25.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:59:25.301 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-55834574851 2026-02-22T22:59:25.301 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:59:25.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=55834574851 2026-02-22T22:59:25.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-22T22:59:25.302 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 55834574851 2026-02-22T22:59:25.303 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:59:25.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-22T22:59:25.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:59:25.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:59:25.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:59:25.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 == 0 2026-02-22T22:59:25.743 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:59:25.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:59:25.744 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:59:25.744 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:59:25.744 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:59:25.744 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:59:25.744 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:59:25.927 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=8 2026-02-22T22:59:25.927 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:59:25.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:59:25.928 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:59:26.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 8 = 8 2026-02-22T22:59:26.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:59:26.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:59:26.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2026-02-22T22:59:26.183 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2026-02-22T22:59:26.184 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2026-02-22T22:59:26.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2026-02-22T22:59:26.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2026-02-22T22:59:26.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2026-02-22T22:59:26.370 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records in 2026-02-22T22:59:26.370 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records out 2026-02-22T22:59:26.370 INFO:tasks.workunit.client.0.trial123.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 3.9673e-05 s, 103 MB/s 2026-02-22T22:59:26.370 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2026-02-22T22:59:26.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2026-02-22T22:59:26.391 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.391 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2026-02-22T22:59:26.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2026-02-22T22:59:26.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2026-02-22T22:59:26.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2026-02-22T22:59:26.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.462 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2026-02-22T22:59:26.479 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.479 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2026-02-22T22:59:26.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2026-02-22T22:59:26.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2026-02-22T22:59:26.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.530 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2026-02-22T22:59:26.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2026-02-22T22:59:26.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2026-02-22T22:59:26.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2026-02-22T22:59:26.613 INFO:tasks.workunit.client.0.trial123.stdout:created pool test snap snap1 2026-02-22T22:59:26.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2026-02-22T22:59:26.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2026-02-22T22:59:26.615 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records in 2026-02-22T22:59:26.615 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records out 2026-02-22T22:59:26.615 INFO:tasks.workunit.client.0.trial123.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.3841e-05 s, 93.4 MB/s 2026-02-22T22:59:26.616 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2026-02-22T22:59:26.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2026-02-22T22:59:26.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2026-02-22T22:59:26.655 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.656 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2026-02-22T22:59:26.674 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.674 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2026-02-22T22:59:26.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.692 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2026-02-22T22:59:26.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2026-02-22T22:59:26.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2026-02-22T22:59:26.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2026-02-22T22:59:26.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.765 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2026-02-22T22:59:26.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-22T22:59:26.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2026-02-22T22:59:26.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2026-02-22T22:59:26.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2026-02-22T22:59:26.916 INFO:tasks.workunit.client.0.trial123.stdout:removed pool test snap snap1 2026-02-22T22:59:26.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2026-02-22T22:59:26.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:59:26.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:59:26.920 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:59:26.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:59:26.921 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:59:26.921 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:59:26.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:59:26.921 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:59:26.922 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:59:27.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:59:27.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:59:27.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:59:27.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:59:27.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:59:27.004 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:59:27.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:59:27.194 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T22:59:27.194 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T22:59:27.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:59:27.194 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:59:27.195 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:59:27.255 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836490 2026-02-22T22:59:27.255 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836490 2026-02-22T22:59:27.255 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836490' 2026-02-22T22:59:27.256 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:59:27.256 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:59:27.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=38654705671 2026-02-22T22:59:27.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 38654705671 2026-02-22T22:59:27.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836490 1-38654705671' 2026-02-22T22:59:27.316 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:59:27.316 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:59:27.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=55834574854 2026-02-22T22:59:27.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 55834574854 2026-02-22T22:59:27.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836490 1-38654705671 2-55834574854' 2026-02-22T22:59:27.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:59:27.377 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836490 2026-02-22T22:59:27.377 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:59:27.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:59:27.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836490 2026-02-22T22:59:27.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:59:27.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836490 2026-02-22T22:59:27.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-22T22:59:27.380 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836490 2026-02-22T22:59:27.380 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:59:27.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836488 -lt 21474836490 2026-02-22T22:59:27.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:59:28.567 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:59:28.567 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:59:28.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836488 -lt 21474836490 2026-02-22T22:59:28.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:59:29.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:59:29.756 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:59:29.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836490 -lt 21474836490 2026-02-22T22:59:29.944 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:59:29.945 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-38654705671 2026-02-22T22:59:29.945 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:59:29.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:59:29.946 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-38654705671 2026-02-22T22:59:29.946 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:59:29.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=38654705671 2026-02-22T22:59:29.948 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-22T22:59:29.948 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 38654705671 2026-02-22T22:59:29.948 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:59:30.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 38654705671 -lt 38654705671 2026-02-22T22:59:30.134 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:59:30.134 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-55834574854 2026-02-22T22:59:30.134 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:59:30.135 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:59:30.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-55834574854 2026-02-22T22:59:30.136 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:59:30.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=55834574854 2026-02-22T22:59:30.137 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 55834574854' 2026-02-22T22:59:30.137 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 55834574854 2026-02-22T22:59:30.137 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:59:30.322 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 55834574854 -lt 55834574854 2026-02-22T22:59:30.323 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:59:30.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:59:30.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:59:30.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 == 0 2026-02-22T22:59:30.573 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:59:30.573 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:59:30.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:59:30.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:59:30.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:59:30.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:59:30.574 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:59:30.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=8 2026-02-22T22:59:30.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:59:30.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:59:30.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:59:31.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 8 = 8 2026-02-22T22:59:31.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:59:31.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:59:31.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2026-02-22T22:59:41.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2026-02-22T22:59:41.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2026-02-22T22:59:41.022 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2026-02-22T22:59:41.024 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2026-02-22T22:59:41.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.025 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2026-02-22T22:59:41.026 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2026-02-22T22:59:41.027 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.089 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2026-02-22T22:59:41.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2026-02-22T22:59:41.091 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.092 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2026-02-22T22:59:41.092 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:41.155 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.001927144 2026-02-22T22:59:41.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.001927144 2026-02-22T22:59:41.156 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.156 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2026-02-22T22:59:41.156 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2026-02-22T22:59:41.156 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.219 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2026-02-22T22:59:41.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2026-02-22T22:59:41.220 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.221 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2026-02-22T22:59:41.221 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:41.284 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .001927144 + 0.001774542 2026-02-22T22:59:41.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.003701686 2026-02-22T22:59:41.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2026-02-22T22:59:41.286 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2026-02-22T22:59:41.286 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.348 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2026-02-22T22:59:41.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2026-02-22T22:59:41.350 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.350 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2026-02-22T22:59:41.350 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:41.413 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .003701686 + 0.00028147099999999999 2026-02-22T22:59:41.414 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.00398315699999999999 2026-02-22T22:59:41.414 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.414 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2026-02-22T22:59:41.415 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2026-02-22T22:59:41.415 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.480 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2026-02-22T22:59:41.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2026-02-22T22:59:41.482 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.482 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2026-02-22T22:59:41.483 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:41.545 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .00398315699999999999 + 0.0019818359999999998 2026-02-22T22:59:41.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.00596499299999999979 2026-02-22T22:59:41.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2026-02-22T22:59:41.547 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2026-02-22T22:59:41.547 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.609 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2026-02-22T22:59:41.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2026-02-22T22:59:41.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.611 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2026-02-22T22:59:41.611 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:41.673 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .00596499299999999979 + 0.0034810980000000002 2026-02-22T22:59:41.674 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.00944609099999999999 2026-02-22T22:59:41.674 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.675 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2026-02-22T22:59:41.675 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2026-02-22T22:59:41.675 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.738 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2026-02-22T22:59:41.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2026-02-22T22:59:41.739 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.740 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2026-02-22T22:59:41.740 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:41.803 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .00944609099999999999 + 0.001595431 2026-02-22T22:59:41.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.01104152199999999999 2026-02-22T22:59:41.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2026-02-22T22:59:41.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2026-02-22T22:59:41.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.868 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2026-02-22T22:59:41.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2026-02-22T22:59:41.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.870 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2026-02-22T22:59:41.870 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:41.932 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .01104152199999999999 + 0.0021370629999999998 2026-02-22T22:59:41.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.01317858499999999979 2026-02-22T22:59:41.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T22:59:41.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2026-02-22T22:59:41.934 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2026-02-22T22:59:41.934 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-22T22:59:41.997 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2026-02-22T22:59:41.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2026-02-22T22:59:41.999 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-22T22:59:41.999 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2026-02-22T22:59:41.999 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-22T22:59:42.063 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .01317858499999999979 + 0.0040302339999999997 2026-02-22T22:59:42.064 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.01720881899999999949 2026-02-22T22:59:42.064 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2026-02-22T22:59:42.064 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '.01720881899999999949 > 0.0' 2026-02-22T22:59:42.065 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2026-02-22T22:59:42.065 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:42.066 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2026-02-22T22:59:42.066 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.066 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:59:42.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T22:59:42.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:59:42.067 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:59:42.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:59:42.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:59:42.067 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:59:42.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:59:42.173 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:59:42.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:59:42.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:59:42.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:59:42.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:59:42.176 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:59:42.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.177 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:59:42.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:59:42.178 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.178 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:59:42.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:59:42.179 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T22:59:42.208 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:59:42.208 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.208 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T22:59:42.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:59:42.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:59:42.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2026-02-22T22:59:42.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:59:42.210 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T22:59:42.211 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:59:42.211 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:59:42.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:59:42.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:59:42.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:59:42.211 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:59:42.212 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:59:42.212 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:59:42.213 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:59:42.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:59:42.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:59:42.214 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:59:42.214 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:59:42.215 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:59:42.215 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.216 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:59:42.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:59:42.217 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T22:59:42.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:59:42.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.218 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.220 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:59:42.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T22:59:42.221 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:59:42.221 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:59:42.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:59:42.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:59:42.221 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:59:42.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:59:42.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:59:42.222 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:59:42.222 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:59:42.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:59:42.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:59:42.223 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:59:42.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:59:42.225 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:59:42.225 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:59:42.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:59:42.226 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T22:59:42.228 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:59:42.228 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.228 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T22:59:42.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:59:42.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:59:42.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2026-02-22T22:59:42.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:59:42.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.175810 2026-02-22T22:59:42.231 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:59:42.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:59:42.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:59:42.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-22T22:59:42.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2026-02-22T22:59:42.233 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:59:42.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T22:59:42.235 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:59:42.235 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:59:42.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:59:42.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:59:42.235 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:59:42.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:59:42.236 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:59:42.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:59:42.236 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:59:42.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:59:42.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:59:42.236 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:59:42.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:59:42.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:59:42.238 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:59:42.238 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:59:42.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:59:42.239 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T22:59:42.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:59:42.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.240 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.240 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T22:59:42.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:59:42.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:59:42.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2026-02-22T22:59:42.242 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:59:42.242 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.242 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.175810 2026-02-22T22:59:42.243 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:59:42.244 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2026-02-22T22:59:42.245 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2026-02-22T22:59:42.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:59:42.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:59:42.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:59:42.264 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:59:42.264 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.264 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.264 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:42.265 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T22:59:42.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:59:42.292 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:59:42.293 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:59:42.293 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.293 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.293 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175810/ceph-mon.a.asok 2026-02-22T22:59:42.293 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:59:42.293 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.175810/ceph-mon.a.asok config get fsid 2026-02-22T22:59:42.337 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:59:42.337 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:59:42.338 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:59:42.338 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:59:42.338 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:59:42.338 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:59:42.338 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:59:42.338 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:59:42.339 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:59:42.339 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.339 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.339 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175810/ceph-mon.a.asok 2026-02-22T22:59:42.339 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:59:42.340 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.175810/ceph-mon.a.asok config get mon_host 2026-02-22T22:59:42.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2026-02-22T22:59:42.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:59:42.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:59:42.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:59:42.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2026-02-22T22:59:42.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:59:42.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:59:42.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:59:42.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:59:42.481 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:59:42.481 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.481 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.482 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:42.482 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:59:42.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:59:42.498 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2026-02-22T22:59:42.499 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2026-02-22T22:59:42.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T22:59:42.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:59:42.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:42.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:42.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2026-02-22T22:59:42.507 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:59:42.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e689a0f-3176-46ba-b6d6-61a7bc1dc0fb 2026-02-22T22:59:42.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5e689a0f-3176-46ba-b6d6-61a7bc1dc0fb' 2026-02-22T22:59:42.507 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 5e689a0f-3176-46ba-b6d6-61a7bc1dc0fb 2026-02-22T22:59:42.507 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:59:42.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeiptp3ft6HhAAd/fpQh3D5sIpu7OF+hN5UA== 2026-02-22T22:59:42.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeiptp3ft6HhAAd/fpQh3D5sIpu7OF+hN5UA=="}' 2026-02-22T22:59:42.513 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e689a0f-3176-46ba-b6d6-61a7bc1dc0fb -i td/test-snaptrim-stats/0/new.json 2026-02-22T22:59:42.614 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:59:42.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2026-02-22T22:59:42.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBeiptp3ft6HhAAd/fpQh3D5sIpu7OF+hN5UA== --osd-uuid 5e689a0f-3176-46ba-b6d6-61a7bc1dc0fb 2026-02-22T22:59:42.639 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:42.637+0000 7f72d65f9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:42.640 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:42.638+0000 7f72d65f9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:42.641 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:42.639+0000 7f72d65f9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:42.641 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:42.639+0000 7f72d65f9780 -1 bdev(0x55965a5ebc00 td/test-snaptrim-stats/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:59:42.641 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:42.639+0000 7f72d65f9780 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2026-02-22T22:59:45.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2026-02-22T22:59:45.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:59:45.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:59:45.301 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T22:59:45.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:59:45.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:59:45.554 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T22:59:45.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:45.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:59:45.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:59:45.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:59:45.570 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:45.568+0000 7f9c70013780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:45.571 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:45.569+0000 7f9c70013780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:45.572 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:45.570+0000 7f9c70013780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:45.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:59:45.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:59:45.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:59:45.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:59:45.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:59:45.738 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:45.738 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:59:45.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:59:45.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:45.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:45.928 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:46.929 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:46.930 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:46.930 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:46.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:59:46.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:46.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:47.114 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:47.163 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:47.161+0000 7f9c70013780 -1 Falling back to public interface 2026-02-22T22:59:48.115 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:48.116 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:48.116 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:59:48.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:59:48.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:48.117 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:48.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:49.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:49.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:49.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:59:49.303 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:59:49.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:49.303 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:49.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:50.102 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:50.100+0000 7f9c70013780 -1 osd.0 0 log_to_monitors true 2026-02-22T22:59:50.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:50.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:50.491 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:59:50.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:59:50.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:50.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:50.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:51.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:51.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:51.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:59:51.687 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T22:59:51.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:51.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:59:51.873 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/648033637,v1:127.0.0.1:6803/648033637] [v2:127.0.0.1:6804/648033637,v1:127.0.0.1:6805/648033637] exists,up 5e689a0f-3176-46ba-b6d6-61a7bc1dc0fb 2026-02-22T22:59:51.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:59:51.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:59:51.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:59:51.877 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2026-02-22T22:59:51.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T22:59:51.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2026-02-22T22:59:51.880 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:59:51.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7165642c-b4e6-4db9-80a7-02ac9b21ea79 2026-02-22T22:59:51.881 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 7165642c-b4e6-4db9-80a7-02ac9b21ea79 2026-02-22T22:59:51.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7165642c-b4e6-4db9-80a7-02ac9b21ea79' 2026-02-22T22:59:51.881 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:59:51.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBniptpBzjQNBAAEeFmgrf8p56GaDjeyLk1zA== 2026-02-22T22:59:51.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBniptpBzjQNBAAEeFmgrf8p56GaDjeyLk1zA=="}' 2026-02-22T22:59:51.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7165642c-b4e6-4db9-80a7-02ac9b21ea79 -i td/test-snaptrim-stats/1/new.json 2026-02-22T22:59:52.079 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:52.086 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2026-02-22T22:59:52.087 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBniptpBzjQNBAAEeFmgrf8p56GaDjeyLk1zA== --osd-uuid 7165642c-b4e6-4db9-80a7-02ac9b21ea79 2026-02-22T22:59:52.105 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:52.103+0000 7f9834c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:52.106 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:52.104+0000 7f9834c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:52.107 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:52.105+0000 7f9834c1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:52.107 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:52.105+0000 7f9834c1c780 -1 bdev(0x55b1fae67c00 td/test-snaptrim-stats/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:59:52.108 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:52.106+0000 7f9834c1c780 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2026-02-22T22:59:54.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2026-02-22T22:59:54.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:59:54.242 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:59:54.242 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T22:59:54.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:59:54.496 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:59:54.497 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T22:59:54.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-22T22:59:54.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:59:54.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:59:54.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:59:54.513 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:54.511+0000 7f0565414780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:54.514 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:54.512+0000 7f0565414780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:54.515 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:54.513+0000 7f0565414780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:59:54.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:59:54.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:59:54.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:59:54.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:59:54.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:59:54.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:54.687 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T22:59:54.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:59:54.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:54.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:54.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:55.872 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:55.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:55.873 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:59:55.873 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T22:59:55.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:55.874 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:56.060 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:56.105 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:56.102+0000 7f0565414780 -1 Falling back to public interface 2026-02-22T22:59:57.061 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:57.062 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:57.062 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T22:59:57.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:59:57.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:57.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:57.248 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:58.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:58.249 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:58.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:59:58.250 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T22:59:58.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:58.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:58.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:59:59.049 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T22:59:59.045+0000 7f0565414780 -1 osd.1 0 log_to_monitors true 2026-02-22T22:59:59.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:59:59.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:59:59.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:59:59.437 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T22:59:59.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:59:59.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:59:59.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:00.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:00.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:00.632 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T23:00:00.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T23:00:00.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:00.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:00:00.823 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/661796801,v1:127.0.0.1:6811/661796801] [v2:127.0.0.1:6812/661796801,v1:127.0.0.1:6813/661796801] exists,up 7165642c-b4e6-4db9-80a7-02ac9b21ea79 2026-02-22T23:00:00.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:00:00.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:00:00.828 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T23:00:00.829 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:00:00.830 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-22T23:00:00.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2026-02-22T23:00:00.831 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:00:00.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=56cdbf3e-fbf4-4bee-bc1b-9958d6b27c6f 2026-02-22T23:00:00.831 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 56cdbf3e-fbf4-4bee-bc1b-9958d6b27c6f' 2026-02-22T23:00:00.831 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 56cdbf3e-fbf4-4bee-bc1b-9958d6b27c6f 2026-02-22T23:00:00.832 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:00:00.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwiptpv/7hMRAAT8ipYDaFkyWjmAz2yHyNQQ== 2026-02-22T23:00:00.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwiptpv/7hMRAAT8ipYDaFkyWjmAz2yHyNQQ=="}' 2026-02-22T23:00:00.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 56cdbf3e-fbf4-4bee-bc1b-9958d6b27c6f -i td/test-snaptrim-stats/2/new.json 2026-02-22T23:00:01.030 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:00:01.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2026-02-22T23:00:01.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBwiptpv/7hMRAAT8ipYDaFkyWjmAz2yHyNQQ== --osd-uuid 56cdbf3e-fbf4-4bee-bc1b-9958d6b27c6f 2026-02-22T23:00:01.056 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:01.053+0000 7fa87822e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:01.057 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:01.054+0000 7fa87822e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:01.058 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:01.055+0000 7fa87822e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:01.058 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:01.055+0000 7fa87822e780 -1 bdev(0x55fd7c83fc00 td/test-snaptrim-stats/2/block) open stat got: (1) Operation not permitted 2026-02-22T23:00:01.058 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:01.055+0000 7fa87822e780 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2026-02-22T23:00:03.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2026-02-22T23:00:03.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:00:03.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T23:00:03.184 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T23:00:03.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:00:03.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T23:00:03.438 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T23:00:03.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=557853ff-66f7-46ac-ae41-abfa5d6ba002 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.175810/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-22T23:00:03.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:00:03.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:00:03.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:00:03.455 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:03.452+0000 7fd1b280c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:03.457 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:03.453+0000 7fd1b280c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:03.457 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:03.454+0000 7fd1b280c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:03.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T23:00:03.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:00:03.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T23:00:03.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:00:03.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:00:03.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:03.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:00:03.627 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:00:03.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:03.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:00:03.811 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:04.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:04.812 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:04.813 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:00:04.813 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:00:04.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:04.814 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:00:04.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:05.043 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:05.039+0000 7fd1b280c780 -1 Falling back to public interface 2026-02-22T23:00:05.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:05.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:05.998 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:00:05.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:00:05.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:05.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:00:06.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:07.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:07.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:07.184 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:00:07.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:00:07.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:07.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:00:07.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:07.738 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:07.736+0000 7fd1b280c780 -1 osd.2 0 log_to_monitors true 2026-02-22T23:00:08.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:08.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:08.373 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:00:08.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:00:08.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:08.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:00:08.565 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1280555896,v1:127.0.0.1:6819/1280555896] [v2:127.0.0.1:6820/1280555896,v1:127.0.0.1:6821/1280555896] exists,up 56cdbf3e-fbf4-4bee-bc1b-9958d6b27c6f 2026-02-22T23:00:08.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:00:08.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:00:08.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:00:08.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2026-02-22T23:00:08.830 INFO:tasks.workunit.client.0.trial123.stderr:noscrub is set 2026-02-22T23:00:08.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2026-02-22T23:00:09.133 INFO:tasks.workunit.client.0.trial123.stderr:nodeep-scrub is set 2026-02-22T23:00:09.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2026-02-22T23:00:09.141 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2026-02-22T23:00:09.388 INFO:tasks.workunit.client.0.trial123.stderr:pool 'test' already exists 2026-02-22T23:00:09.395 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T23:00:10.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2026-02-22T23:00:10.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T23:00:10.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T23:00:10.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T23:00:10.397 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T23:00:10.398 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T23:00:10.399 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T23:00:10.399 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T23:00:10.399 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T23:00:10.399 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T23:00:10.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T23:00:10.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T23:00:10.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T23:00:10.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T23:00:10.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T23:00:10.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T23:00:10.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T23:00:10.669 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T23:00:10.669 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T23:00:10.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T23:00:10.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:10.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T23:00:10.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836486 2026-02-22T23:00:10.732 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836486 2026-02-22T23:00:10.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T23:00:10.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:10.733 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T23:00:10.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672965 2026-02-22T23:00:10.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672965 2026-02-22T23:00:10.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2026-02-22T23:00:10.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:10.792 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T23:00:10.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509444 2026-02-22T23:00:10.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509444 2026-02-22T23:00:10.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2026-02-22T23:00:10.854 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:10.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836486 2026-02-22T23:00:10.855 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:10.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T23:00:10.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836486 2026-02-22T23:00:10.856 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:10.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836486 2026-02-22T23:00:10.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T23:00:10.858 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836486 2026-02-22T23:00:10.858 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T23:00:11.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-22T23:00:11.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T23:00:12.046 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T23:00:12.046 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T23:00:12.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-22T23:00:12.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T23:00:13.234 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T23:00:13.235 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T23:00:13.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836487 -lt 21474836486 2026-02-22T23:00:13.423 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:13.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672965 2026-02-22T23:00:13.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:13.425 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T23:00:13.425 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672965 2026-02-22T23:00:13.425 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:13.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672965 2026-02-22T23:00:13.427 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-22T23:00:13.427 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672965 2026-02-22T23:00:13.427 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T23:00:13.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-22T23:00:13.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:13.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509444 2026-02-22T23:00:13.612 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:13.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T23:00:13.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509444 2026-02-22T23:00:13.614 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:13.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509444 2026-02-22T23:00:13.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-22T23:00:13.615 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509444 2026-02-22T23:00:13.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T23:00:13.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509444 -lt 64424509444 2026-02-22T23:00:13.799 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T23:00:13.800 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T23:00:13.800 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T23:00:14.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 == 0 2026-02-22T23:00:14.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T23:00:14.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T23:00:14.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T23:00:14.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T23:00:14.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T23:00:14.070 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T23:00:14.071 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T23:00:14.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=8 2026-02-22T23:00:14.252 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T23:00:14.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T23:00:14.253 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T23:00:14.505 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 8 = 8 2026-02-22T23:00:14.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T23:00:14.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T23:00:14.506 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2026-02-22T23:00:14.506 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2026-02-22T23:00:14.506 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2026-02-22T23:00:14.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2026-02-22T23:00:14.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2026-02-22T23:00:14.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2026-02-22T23:00:14.696 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records in 2026-02-22T23:00:14.696 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records out 2026-02-22T23:00:14.696 INFO:tasks.workunit.client.0.trial123.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.2619e-05 s, 96.1 MB/s 2026-02-22T23:00:14.697 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2026-02-22T23:00:14.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2026-02-22T23:00:14.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.716 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2026-02-22T23:00:14.733 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.734 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2026-02-22T23:00:14.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2026-02-22T23:00:14.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.769 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2026-02-22T23:00:14.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.787 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2026-02-22T23:00:14.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2026-02-22T23:00:14.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.821 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2026-02-22T23:00:14.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2026-02-22T23:00:14.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2026-02-22T23:00:14.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2026-02-22T23:00:14.875 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2026-02-22T23:00:14.876 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2026-02-22T23:00:14.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-22T23:00:14.876 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2026-02-22T23:00:14.944 INFO:tasks.workunit.client.0.trial123.stdout:created pool test snap snap1 2026-02-22T23:00:14.945 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2026-02-22T23:00:14.946 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2026-02-22T23:00:14.946 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records in 2026-02-22T23:00:14.946 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records out 2026-02-22T23:00:14.946 INFO:tasks.workunit.client.0.trial123.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.0655e-05 s, 101 MB/s 2026-02-22T23:00:14.947 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2026-02-22T23:00:14.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.947 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2026-02-22T23:00:14.966 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.966 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2026-02-22T23:00:14.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:14.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2026-02-22T23:00:15.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.003 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2026-02-22T23:00:15.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.021 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2026-02-22T23:00:15.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2026-02-22T23:00:15.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2026-02-22T23:00:15.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2026-02-22T23:00:15.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2026-02-22T23:00:15.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.108 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2026-02-22T23:00:15.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2026-02-22T23:00:15.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-22T23:00:15.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2026-02-22T23:00:15.145 INFO:tasks.workunit.client.0.trial123.stdout:created pool test snap snap2 2026-02-22T23:00:15.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2026-02-22T23:00:15.146 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2026-02-22T23:00:15.147 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records in 2026-02-22T23:00:15.147 INFO:tasks.workunit.client.0.trial123.stderr:1+0 records out 2026-02-22T23:00:15.147 INFO:tasks.workunit.client.0.trial123.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.4232e-05 s, 92.6 MB/s 2026-02-22T23:00:15.147 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2026-02-22T23:00:15.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.148 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2026-02-22T23:00:15.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2026-02-22T23:00:15.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.184 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2026-02-22T23:00:15.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.201 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2026-02-22T23:00:15.218 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.219 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2026-02-22T23:00:15.236 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.237 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2026-02-22T23:00:15.254 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.254 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2026-02-22T23:00:15.271 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2026-02-22T23:00:15.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.290 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2026-02-22T23:00:15.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-22T23:00:15.308 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2026-02-22T23:00:15.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2026-02-22T23:00:15.326 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2026-02-22T23:00:15.326 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2026-02-22T23:00:15.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-22T23:00:15.327 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2026-02-22T23:00:15.345 INFO:tasks.workunit.client.0.trial123.stdout:removed pool test snap snap1 2026-02-22T23:00:15.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2026-02-22T23:00:15.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T23:00:15.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T23:00:15.347 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T23:00:15.348 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T23:00:15.348 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T23:00:15.349 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T23:00:15.349 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T23:00:15.349 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T23:00:15.349 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T23:00:15.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T23:00:15.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T23:00:15.431 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T23:00:15.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T23:00:15.432 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T23:00:15.432 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T23:00:15.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T23:00:15.616 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T23:00:15.617 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T23:00:15.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T23:00:15.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:15.617 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T23:00:15.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836489 2026-02-22T23:00:15.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836489 2026-02-22T23:00:15.676 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836489' 2026-02-22T23:00:15.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:15.677 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T23:00:15.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672968 2026-02-22T23:00:15.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672968 2026-02-22T23:00:15.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2026-02-22T23:00:15.737 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:15.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T23:00:15.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509447 2026-02-22T23:00:15.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509447 2026-02-22T23:00:15.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2026-02-22T23:00:15.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:15.796 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836489 2026-02-22T23:00:15.796 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:15.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T23:00:15.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836489 2026-02-22T23:00:15.798 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:15.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836489 2026-02-22T23:00:15.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-22T23:00:15.799 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836489 2026-02-22T23:00:15.800 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T23:00:15.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-22T23:00:15.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T23:00:16.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T23:00:16.986 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T23:00:17.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836490 -lt 21474836489 2026-02-22T23:00:17.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:17.173 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672968 2026-02-22T23:00:17.174 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:17.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T23:00:17.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672968 2026-02-22T23:00:17.175 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:17.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672968 2026-02-22T23:00:17.176 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-22T23:00:17.176 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672968 2026-02-22T23:00:17.177 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T23:00:17.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-22T23:00:17.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:17.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509447 2026-02-22T23:00:17.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:17.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T23:00:17.366 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509447 2026-02-22T23:00:17.366 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:17.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509447 2026-02-22T23:00:17.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-22T23:00:17.367 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509447 2026-02-22T23:00:17.367 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T23:00:17.555 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-22T23:00:17.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T23:00:17.556 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T23:00:17.557 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T23:00:17.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 == 0 2026-02-22T23:00:17.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T23:00:17.807 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T23:00:17.807 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T23:00:17.807 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T23:00:17.807 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T23:00:17.807 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T23:00:17.808 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T23:00:17.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=8 2026-02-22T23:00:17.990 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T23:00:17.990 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T23:00:17.990 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T23:00:18.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 8 = 8 2026-02-22T23:00:18.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T23:00:18.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T23:00:18.243 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2026-02-22T23:00:28.244 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2026-02-22T23:00:28.244 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2026-02-22T23:00:28.245 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2026-02-22T23:00:28.246 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2026-02-22T23:00:28.247 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:28.247 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2026-02-22T23:00:28.248 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-22T23:00:28.248 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:28.314 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2026-02-22T23:00:28.315 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2026-02-22T23:00:28.315 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:28.316 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-22T23:00:28.316 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:28.377 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.001846596 2026-02-22T23:00:28.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.001846596 2026-02-22T23:00:28.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:28.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2026-02-22T23:00:28.379 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-22T23:00:28.379 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:28.443 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2026-02-22T23:00:28.444 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-22T23:00:28.444 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:28.445 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-22T23:00:28.445 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:28.509 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .001846596 + 0.001796483 2026-02-22T23:00:28.510 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.003643079 2026-02-22T23:00:28.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:28.511 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2026-02-22T23:00:28.511 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-22T23:00:28.511 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:28.575 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2026-02-22T23:00:28.576 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-22T23:00:28.577 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:28.577 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-22T23:00:28.578 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:28.642 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .003643079 + 0.00019916899999999999 2026-02-22T23:00:28.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.00384224799999999999 2026-02-22T23:00:28.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:28.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2026-02-22T23:00:28.644 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-22T23:00:28.644 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:28.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2026-02-22T23:00:28.709 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2026-02-22T23:00:28.709 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:28.710 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-22T23:00:28.710 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:28.773 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .00384224799999999999 + 0.001921475 2026-02-22T23:00:28.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.00576372299999999999 2026-02-22T23:00:28.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:28.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2026-02-22T23:00:28.774 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-22T23:00:28.774 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:28.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2026-02-22T23:00:28.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2026-02-22T23:00:28.837 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:28.838 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-22T23:00:28.838 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:28.901 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .00576372299999999999 + 0.0032872510000000001 2026-02-22T23:00:28.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.00905097400000000009 2026-02-22T23:00:28.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:28.902 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2026-02-22T23:00:28.903 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-22T23:00:28.903 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:28.966 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2026-02-22T23:00:28.967 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-22T23:00:28.968 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:28.968 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-22T23:00:28.969 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:29.035 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .00905097400000000009 + 0.001027371 2026-02-22T23:00:29.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.01007834500000000009 2026-02-22T23:00:29.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:29.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2026-02-22T23:00:29.036 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-22T23:00:29.037 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:29.100 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2026-02-22T23:00:29.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-22T23:00:29.101 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:29.102 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-22T23:00:29.102 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:29.166 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .01007834500000000009 + 0.0020377490000000002 2026-02-22T23:00:29.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.01211609400000000029 2026-02-22T23:00:29.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:29.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2026-02-22T23:00:29.168 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-22T23:00:29.168 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:29.231 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2026-02-22T23:00:29.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2026-02-22T23:00:29.233 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:29.233 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-22T23:00:29.233 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:29.296 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .01211609400000000029 + 0.0033015470000000002 2026-02-22T23:00:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.01541764100000000049 2026-02-22T23:00:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2026-02-22T23:00:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '.01541764100000000049 > 0.0' 2026-02-22T23:00:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:29.298 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:00:29.298 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2026-02-22T23:00:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2026-02-22T23:00:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-22T23:00:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2026-02-22T23:00:29.368 INFO:tasks.workunit.client.0.trial123.stdout:removed pool test snap snap2 2026-02-22T23:00:29.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2026-02-22T23:00:29.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T23:00:29.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T23:00:29.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T23:00:29.371 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T23:00:29.371 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T23:00:29.371 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T23:00:29.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T23:00:29.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T23:00:29.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T23:00:29.453 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T23:00:29.453 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T23:00:29.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T23:00:29.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T23:00:29.454 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T23:00:29.454 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T23:00:29.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T23:00:29.639 INFO:tasks.workunit.client.0.trial123.stderr:1 2026-02-22T23:00:29.639 INFO:tasks.workunit.client.0.trial123.stderr:2' 2026-02-22T23:00:29.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T23:00:29.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:29.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T23:00:29.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836494 2026-02-22T23:00:29.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836494 2026-02-22T23:00:29.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836494' 2026-02-22T23:00:29.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:29.702 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T23:00:29.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672972 2026-02-22T23:00:29.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672972 2026-02-22T23:00:29.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2026-02-22T23:00:29.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T23:00:29.763 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T23:00:29.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=64424509452 2026-02-22T23:00:29.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 64424509452 2026-02-22T23:00:29.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509452' 2026-02-22T23:00:29.824 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:29.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836494 2026-02-22T23:00:29.824 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:29.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T23:00:29.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836494 2026-02-22T23:00:29.826 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:29.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836494 2026-02-22T23:00:29.827 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2026-02-22T23:00:29.828 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.0 seq 21474836494 2026-02-22T23:00:29.828 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T23:00:30.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836492 -lt 21474836494 2026-02-22T23:00:30.013 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T23:00:31.014 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T23:00:31.015 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T23:00:31.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836494 -lt 21474836494 2026-02-22T23:00:31.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:31.202 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672972 2026-02-22T23:00:31.202 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:31.202 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T23:00:31.203 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672972 2026-02-22T23:00:31.203 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:31.204 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672972 2026-02-22T23:00:31.205 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2026-02-22T23:00:31.205 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.1 seq 42949672972 2026-02-22T23:00:31.205 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T23:00:31.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672973 -lt 42949672972 2026-02-22T23:00:31.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T23:00:31.394 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-64424509452 2026-02-22T23:00:31.394 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T23:00:31.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T23:00:31.395 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-64424509452 2026-02-22T23:00:31.395 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T23:00:31.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=64424509452 2026-02-22T23:00:31.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2026-02-22T23:00:31.396 INFO:tasks.workunit.client.0.trial123.stdout:waiting osd.2 seq 64424509452 2026-02-22T23:00:31.397 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T23:00:31.583 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 64424509452 -lt 64424509452 2026-02-22T23:00:31.583 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T23:00:31.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T23:00:31.584 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T23:00:31.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 == 0 2026-02-22T23:00:31.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T23:00:31.835 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T23:00:31.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T23:00:31.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T23:00:31.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T23:00:31.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T23:00:31.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T23:00:32.020 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=8 2026-02-22T23:00:32.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T23:00:32.020 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T23:00:32.021 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T23:00:32.271 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 8 = 8 2026-02-22T23:00:32.271 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T23:00:32.271 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T23:00:32.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2026-02-22T23:00:42.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2026-02-22T23:00:42.273 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2026-02-22T23:00:42.273 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2026-02-22T23:00:42.274 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2026-02-22T23:00:42.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:42.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2026-02-22T23:00:42.276 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-22T23:00:42.276 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:42.341 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2026-02-22T23:00:42.342 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2026-02-22T23:00:42.343 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:42.343 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-22T23:00:42.343 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:42.405 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.002030025 2026-02-22T23:00:42.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.002030025 2026-02-22T23:00:42.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:42.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2026-02-22T23:00:42.407 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-22T23:00:42.407 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:42.469 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2026-02-22T23:00:42.470 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-22T23:00:42.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:42.472 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-22T23:00:42.472 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:42.535 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .002030025 + 0.001812092 2026-02-22T23:00:42.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.003842117 2026-02-22T23:00:42.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:42.536 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2026-02-22T23:00:42.537 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-22T23:00:42.537 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:42.602 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2026-02-22T23:00:42.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-22T23:00:42.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:42.604 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-22T23:00:42.604 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:42.669 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .003842117 + 0.00022156000000000001 2026-02-22T23:00:42.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.00406367700000000001 2026-02-22T23:00:42.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:42.670 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2026-02-22T23:00:42.671 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-22T23:00:42.671 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:42.735 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2026-02-22T23:00:42.736 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2026-02-22T23:00:42.737 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:42.737 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-22T23:00:42.738 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:42.800 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .00406367700000000001 + 0.00176249 2026-02-22T23:00:42.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.00582616700000000001 2026-02-22T23:00:42.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:42.803 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2026-02-22T23:00:42.803 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-22T23:00:42.804 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:42.867 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2026-02-22T23:00:42.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2026-02-22T23:00:42.869 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:42.869 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-22T23:00:42.869 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:42.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .00582616700000000001 + 0.0031930169999999998 2026-02-22T23:00:42.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.00901918399999999981 2026-02-22T23:00:42.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:42.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2026-02-22T23:00:42.935 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-22T23:00:42.935 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:42.997 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2026-02-22T23:00:42.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-22T23:00:42.999 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:42.999 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-22T23:00:43.000 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:43.062 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .00901918399999999981 + 0.001508751 2026-02-22T23:00:43.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.01052793499999999981 2026-02-22T23:00:43.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:43.063 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2026-02-22T23:00:43.063 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-22T23:00:43.064 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:43.127 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2026-02-22T23:00:43.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-22T23:00:43.129 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:43.129 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-22T23:00:43.129 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:43.192 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .01052793499999999981 + 0.00036604699999999999 2026-02-22T23:00:43.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.01089398199999999980 2026-02-22T23:00:43.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-22T23:00:43.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2026-02-22T23:00:43.193 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-22T23:00:43.193 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-22T23:00:43.259 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2026-02-22T23:00:43.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2026-02-22T23:00:43.260 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:43.261 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-22T23:00:43.261 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-22T23:00:43.324 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .01089398199999999980 + 0.0029564790000000001 2026-02-22T23:00:43.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.01385046099999999990 2026-02-22T23:00:43.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2026-02-22T23:00:43.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '.01385046099999999990 > 0.0' 2026-02-22T23:00:43.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-22T23:00:43.326 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:00:43.326 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2026-02-22T23:00:43.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2026-02-22T23:00:43.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2026-02-22T23:00:43.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2026-02-22T23:00:43.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T23:00:43.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:00:43.328 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T23:00:43.328 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:00:43.329 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:00:43.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:00:43.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:00:43.329 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:00:43.435 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:00:43.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:00:43.436 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:00:43.436 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:00:43.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:00:43.437 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:00:43.437 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:00:43.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.438 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:00:43.439 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:00:43.439 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:00:43.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:00:43.440 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T23:00:43.469 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:00:43.469 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.469 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T23:00:43.469 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T23:00:43.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:00:43.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:00:43.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2026-02-22T23:00:43.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T23:00:43.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:00:43.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T23:00:43.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:00:43.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:00:43.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:00:43.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:00:43.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:00:43.472 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:00:43.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:00:43.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:00:43.473 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:00:43.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:00:43.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:00:43.475 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:00:43.476 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.477 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:00:43.477 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:00:43.477 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.477 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:00:43.478 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:00:43.478 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T23:00:43.479 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:00:43.479 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.479 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T23:00:43.479 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/test-snaptrim-stats 0 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-22T23:00:43.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:00:43.482 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:00:43.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:00:43.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:00:43.482 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:00:43.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:00:43.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:00:43.483 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:00:43.483 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:00:43.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:00:43.484 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:00:43.484 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:00:43.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.485 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:00:43.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:00:43.486 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.486 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:00:43.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T23:00:43.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2026-02-22T23:00:43.488 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:00:43.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.489 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175810 2026-02-22T23:00:43.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175810 2026-02-22T23:00:43.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:00:43.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:00:43.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T23:00:43.491 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T23:00:43.491 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T23:00:43.556 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2026-02-22T23:00:43.556 DEBUG:teuthology.orchestra.run.trial123:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2026-02-22T23:00:43.620 INFO:tasks.workunit.client.0.trial123.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T23:00:43.624 INFO:tasks.workunit.client.0.trial123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T23:00:43.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/ver-health 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none ' 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2026-02-22T23:00:43.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2026-02-22T23:00:43.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2026-02-22T23:00:43.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2026-02-22T23:00:43.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T23:00:43.628 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:TEST_check_version_health_2 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:TEST_check_version_health_3' 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:00:43.629 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:00:43.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:00:43.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:00:43.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:00:43.630 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:00:43.630 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:00:43.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:00:43.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:00:43.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:00:43.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:00:43.631 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:00:43.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.631 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:00:43.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:00:43.632 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.632 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:00:43.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:00:43.633 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:00:43.634 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:00:43.634 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.635 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:00:43.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:00:43.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:00:43.635 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2026-02-22T23:00:43.636 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T23:00:43.636 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.637 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.637 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189625 2026-02-22T23:00:43.639 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T23:00:43.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T23:00:43.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T23:00:43.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:00:43.640 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:00:43.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:00:43.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:00:43.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:00:43.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:00:43.642 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:00:43.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:00:43.642 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:00:43.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:00:43.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:00:43.643 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:00:43.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:00:43.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:00:43.644 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:00:43.644 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:00:43.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:00:43.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:00:43.646 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:00:43.646 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.646 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.646 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:00:43.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:00:43.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:00:43.647 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2026-02-22T23:00:43.648 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T23:00:43.648 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.648 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.648 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189625 2026-02-22T23:00:43.649 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T23:00:43.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T23:00:43.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T23:00:43.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-22T23:00:43.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-22T23:00:43.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2026-02-22T23:00:43.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T23:00:43.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T23:00:43.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T23:00:43.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2026-02-22T23:00:43.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-22T23:00:43.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T23:00:43.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:00:43.670 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:00:43.670 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:00:43.670 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.670 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.671 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:00:43.671 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-22T23:00:43.696 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T23:00:43.698 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:00:43.699 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.699 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.699 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.a.asok 2026-02-22T23:00:43.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:00:43.699 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.a.asok config get fsid 2026-02-22T23:00:43.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T23:00:43.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:00:43.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T23:00:43.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T23:00:43.743 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T23:00:43.743 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T23:00:43.744 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T23:00:43.744 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T23:00:43.744 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:00:43.744 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.744 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.744 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.a.asok 2026-02-22T23:00:43.745 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:00:43.745 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.a.asok config get mon_host 2026-02-22T23:00:43.790 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-22T23:00:43.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2026-02-22T23:00:43.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T23:00:43.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T23:00:43.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T23:00:43.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2026-02-22T23:00:43.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-22T23:00:43.809 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T23:00:43.809 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:00:43.809 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:00:43.810 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:00:43.810 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.810 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.810 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:00:43.810 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-22T23:00:43.835 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T23:00:43.836 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.837 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:00:43.838 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:00:43.838 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get fsid 2026-02-22T23:00:43.883 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T23:00:43.883 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:00:43.883 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:00:43.884 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get mon_host 2026-02-22T23:00:43.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2026-02-22T23:00:43.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:00:43.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:00:43.931 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T23:00:43.932 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:00:43.933 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=726f6803-5366-42e8-bd23-425e07cdf22a 2026-02-22T23:00:43.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 726f6803-5366-42e8-bd23-425e07cdf22a' 2026-02-22T23:00:43.935 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 726f6803-5366-42e8-bd23-425e07cdf22a 2026-02-22T23:00:43.935 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:00:43.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbiptpX9oOOBAADqwwGwg2bVVEL0jHAdcOwg== 2026-02-22T23:00:43.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbiptpX9oOOBAADqwwGwg2bVVEL0jHAdcOwg=="}' 2026-02-22T23:00:43.943 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 726f6803-5366-42e8-bd23-425e07cdf22a -i td/ver-health/0/new.json 2026-02-22T23:00:50.045 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:00:50.051 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2026-02-22T23:00:50.052 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbiptpX9oOOBAADqwwGwg2bVVEL0jHAdcOwg== --osd-uuid 726f6803-5366-42e8-bd23-425e07cdf22a 2026-02-22T23:00:50.069 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:50.067+0000 7f58f7282780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:50.070 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:50.069+0000 7f58f7282780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:50.071 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:50.070+0000 7f58f7282780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:50.072 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:50.070+0000 7f58f7282780 -1 bdev(0x560b778b9c00 td/ver-health/0/block) open stat got: (1) Operation not permitted 2026-02-22T23:00:50.072 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:50.070+0000 7f58f7282780 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2026-02-22T23:00:52.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2026-02-22T23:00:52.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:00:52.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T23:00:52.198 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T23:00:52.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:00:52.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T23:00:52.300 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T23:00:52.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:00:52.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:00:52.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:00:52.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:00:52.316 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:52.314+0000 7fb0fc010780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:52.318 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:52.316+0000 7fb0fc010780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:52.318 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:52.317+0000 7fb0fc010780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:52.395 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T23:00:52.395 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:00:52.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T23:00:52.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:00:52.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:00:52.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:52.396 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:00:52.396 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:00:52.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:52.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:00:52.491 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:53.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:53.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:53.492 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:00:53.493 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:00:53.493 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:53.494 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:00:53.588 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:53.662 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:53.660+0000 7fb0fc010780 -1 Falling back to public interface 2026-02-22T23:00:54.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:54.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:54.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:00:54.590 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:00:54.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:54.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:00:54.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:55.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:55.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:55.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:00:55.687 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:00:55.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:55.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:00:55.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:56.362 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:56.360+0000 7fb0fc010780 -1 osd.0 0 log_to_monitors true 2026-02-22T23:00:56.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:56.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:56.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:00:56.785 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:00:56.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:56.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:00:56.881 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:00:57.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:00:57.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:00:57.883 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T23:00:57.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T23:00:57.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:00:57.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:00:57.980 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1984054265,v1:127.0.0.1:6801/1984054265] [v2:127.0.0.1:6802/1984054265,v1:127.0.0.1:6803/1984054265] exists,up 726f6803-5366-42e8-bd23-425e07cdf22a 2026-02-22T23:00:57.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:00:57.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:00:57.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:00:57.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2026-02-22T23:00:57.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:00:57.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:00:57.985 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T23:00:57.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:00:57.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2026-02-22T23:00:57.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:00:57.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:00:57.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:00:57.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:00:57.986 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:00:57.987 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2026-02-22T23:00:57.988 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:00:57.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8241c7ed-b37e-449e-9aa1-c820e27d1ef4 2026-02-22T23:00:57.989 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8241c7ed-b37e-449e-9aa1-c820e27d1ef4' 2026-02-22T23:00:57.989 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 8241c7ed-b37e-449e-9aa1-c820e27d1ef4 2026-02-22T23:00:57.989 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:00:57.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpiptpaG0lOxAAR2XGE/AkulMjGM98bUCiKQ== 2026-02-22T23:00:57.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpiptpaG0lOxAAR2XGE/AkulMjGM98bUCiKQ=="}' 2026-02-22T23:00:57.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8241c7ed-b37e-449e-9aa1-c820e27d1ef4 -i td/ver-health/1/new.json 2026-02-22T23:00:58.095 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:00:58.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2026-02-22T23:00:58.101 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpiptpaG0lOxAAR2XGE/AkulMjGM98bUCiKQ== --osd-uuid 8241c7ed-b37e-449e-9aa1-c820e27d1ef4 2026-02-22T23:00:58.119 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:58.117+0000 7fdcbf33d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:58.120 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:58.118+0000 7fdcbf33d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:58.121 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:58.119+0000 7fdcbf33d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:00:58.122 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:58.120+0000 7fdcbf33d780 -1 bdev(0x56453d03dc00 td/ver-health/1/block) open stat got: (1) Operation not permitted 2026-02-22T23:00:58.122 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:00:58.120+0000 7fdcbf33d780 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2026-02-22T23:01:00.250 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2026-02-22T23:01:00.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:01:00.251 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T23:01:00.252 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T23:01:00.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:01:00.353 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T23:01:00.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T23:01:00.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:00.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:01:00.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:01:00.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:01:00.368 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:00.366+0000 7fefece1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:00.369 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:00.368+0000 7fefece1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:00.370 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:00.368+0000 7fefece1c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:00.449 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T23:01:00.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:00.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T23:01:00.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:00.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:00.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:00.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:00.451 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:00.451 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:00.452 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:00.544 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:01.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:01.545 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:01.546 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:01.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:01:01.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:01.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:01.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:01.705 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:01.702+0000 7fefece1c780 -1 Falling back to public interface 2026-02-22T23:01:02.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:02.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:02.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:01:02.642 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:02.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:02.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:02.739 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:03.740 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:03.741 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:03.741 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:01:03.742 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:01:03.742 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:03.742 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:03.837 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:04.640 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:04.638+0000 7fefece1c780 -1 osd.1 0 log_to_monitors true 2026-02-22T23:01:04.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:04.838 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:04.839 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:01:04.839 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:01:04.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:04.840 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:04.934 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:05.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:05.935 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:05.935 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T23:01:05.936 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T23:01:05.936 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:05.936 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:06.032 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2547326012,v1:127.0.0.1:6809/2547326012] [v2:127.0.0.1:6810/2547326012,v1:127.0.0.1:6811/2547326012] exists,up 8241c7ed-b37e-449e-9aa1-c820e27d1ef4 2026-02-22T23:01:06.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2026-02-22T23:01:06.037 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:06.038 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:01:06.039 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb9f443f-ab9f-4f71-9659-9d2e04c3aa7c 2026-02-22T23:01:06.040 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cb9f443f-ab9f-4f71-9659-9d2e04c3aa7c' 2026-02-22T23:01:06.041 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 cb9f443f-ab9f-4f71-9659-9d2e04c3aa7c 2026-02-22T23:01:06.041 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:01:06.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyiptpeGa6AhAAQeMHQVmdAoDS7TsVhEMg2Q== 2026-02-22T23:01:06.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyiptpeGa6AhAAQeMHQVmdAoDS7TsVhEMg2Q=="}' 2026-02-22T23:01:06.047 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb9f443f-ab9f-4f71-9659-9d2e04c3aa7c -i td/ver-health/2/new.json 2026-02-22T23:01:06.148 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:06.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2026-02-22T23:01:06.155 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyiptpeGa6AhAAQeMHQVmdAoDS7TsVhEMg2Q== --osd-uuid cb9f443f-ab9f-4f71-9659-9d2e04c3aa7c 2026-02-22T23:01:06.173 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:06.171+0000 7f156a21f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:06.174 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:06.172+0000 7f156a21f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:06.175 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:06.173+0000 7f156a21f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:06.175 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:06.173+0000 7f156a21f780 -1 bdev(0x555ffc18dc00 td/ver-health/2/block) open stat got: (1) Operation not permitted 2026-02-22T23:01:06.176 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:06.173+0000 7f156a21f780 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2026-02-22T23:01:08.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2026-02-22T23:01:08.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:01:08.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T23:01:08.299 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T23:01:08.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:01:08.398 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T23:01:08.399 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T23:01:08.399 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:08.399 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:01:08.399 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:01:08.400 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:01:08.415 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:08.412+0000 7fe1f882d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:08.416 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:08.413+0000 7fe1f882d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:08.417 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:08.414+0000 7fe1f882d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:08.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T23:01:08.497 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:08.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T23:01:08.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:08.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:08.498 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:08.498 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:08.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:08.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:08.499 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:08.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:09.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:09.596 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:09.597 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:09.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:01:09.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:09.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:09.694 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:09.991 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:09.988+0000 7fe1f882d780 -1 Falling back to public interface 2026-02-22T23:01:10.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:10.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:10.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:01:10.695 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:10.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:10.695 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:10.793 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:11.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:11.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:11.795 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:01:11.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:01:11.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:11.796 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:11.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:12.895 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:12.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:12.896 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:01:12.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:01:12.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:12.897 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:12.963 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:12.959+0000 7fe1f882d780 -1 osd.2 0 log_to_monitors true 2026-02-22T23:01:12.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:13.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:13.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:13.995 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T23:01:13.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T23:01:13.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:13.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:14.096 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/920698007,v1:127.0.0.1:6817/920698007] [v2:127.0.0.1:6818/920698007,v1:127.0.0.1:6819/920698007] exists,up cb9f443f-ab9f-4f71-9659-9d2e04c3aa7c 2026-02-22T23:01:14.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:01:14.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:01:14.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:01:14.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2026-02-22T23:01:19.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:19.191 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_OK 2026-02-22T23:01:19.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:19.199 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2026-02-22T23:01:19.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2026-02-22T23:01:19.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:01:19.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:01:19.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:01:19.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:01:19.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:01:19.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:01:19.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-22T23:01:19.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2026-02-22T23:01:19.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:01:19.406 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:19.407 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:19.408 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:19.408 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:19.408 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:19.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:19.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:01:19.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:01:19.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T23:01:19.410 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T23:01:19.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:19.411 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2026-02-22T23:01:19.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T23:01:19.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:01:19.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:01:19.411 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:01:19.423 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:19.421+0000 7fe79d897780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:19.424 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:19.422+0000 7fe79d897780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:19.425 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:19.423+0000 7fe79d897780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:19.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T23:01:19.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:19.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T23:01:19.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:19.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:19.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:19.507 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:19.508 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:19.508 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:19.509 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:19.602 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:20.517 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:20.515+0000 7fe79d897780 -1 Falling back to public interface 2026-02-22T23:01:20.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:20.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:20.604 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:20.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:01:20.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:20.605 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:20.700 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:21.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:21.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:21.702 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:01:21.702 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:21.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:21.703 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:21.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:22.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:22.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:22.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:01:22.799 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:01:22.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:22.799 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:22.896 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:23.231 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:23.229+0000 7fe79d897780 -1 osd.1 16 log_to_monitors true 2026-02-22T23:01:23.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:23.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:23.898 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:01:23.898 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:01:23.899 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:23.899 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:23.994 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6808/2178133338,v1:127.0.0.1:6809/2178133338] [v2:127.0.0.1:6810/2178133338,v1:127.0.0.1:6811/2178133338] exists,up 8241c7ed-b37e-449e-9aa1-c820e27d1ef4 2026-02-22T23:01:23.995 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:01:23.996 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:01:23.997 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:01:24.093 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-22T23:01:24.093 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-22T23:01:24.093 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-22T23:01:24.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-22T23:01:24.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-22T23:01:24.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:24.187 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2026-02-22T23:01:24.187 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2026-02-22T23:01:24.187 INFO:tasks.workunit.client.0.trial123.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2026-02-22T23:01:24.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:24.193 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:01:24.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:24.293 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2026-02-22T23:01:24.391 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:24.391 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2026-02-22T23:01:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2026-02-22T23:01:24.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:01:24.490 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:01:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:01:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:01:24.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:01:24.598 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:24.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:24.600 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:01:24.601 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2026-02-22T23:01:24.602 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T23:01:24.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T23:01:24.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:24.603 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2026-02-22T23:01:24.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T23:01:24.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:01:24.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:01:24.603 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:01:24.617 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:24.615+0000 7fe190591780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:24.618 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:24.616+0000 7fe190591780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:24.619 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:24.617+0000 7fe190591780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:24.697 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T23:01:24.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:24.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T23:01:24.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:24.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:24.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:24.698 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:24.698 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:24.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:24.699 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:24.792 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:25.710 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:25.708+0000 7fe190591780 -1 Falling back to public interface 2026-02-22T23:01:25.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:25.794 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:25.794 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:25.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:01:25.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:25.795 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:25.892 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:26.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:26.893 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:26.893 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:26.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:01:26.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:26.894 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:26.991 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:27.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:27.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:27.993 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:01:27.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:01:27.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:27.994 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:28.089 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:28.430 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:28.428+0000 7fe190591780 -1 osd.2 19 log_to_monitors true 2026-02-22T23:01:29.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:29.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:29.090 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:01:29.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:01:29.090 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:29.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:01:29.187 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6816/3517310793,v1:127.0.0.1:6817/3517310793] [v2:127.0.0.1:6818/3517310793,v1:127.0.0.1:6819/3517310793] exists,up cb9f443f-ab9f-4f71-9659-9d2e04c3aa7c 2026-02-22T23:01:29.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:01:29.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:01:29.188 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:01:29.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2026-02-22T23:01:29.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:01:29.189 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:01:29.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:01:29.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:01:29.189 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:01:29.296 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2026-02-22T23:01:29.297 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:01:29.298 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:01:29.299 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:01:29.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:01:29.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:01:29.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:01:29.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2026-02-22T23:01:29.300 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T23:01:29.300 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T23:01:29.301 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:29.301 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2026-02-22T23:01:29.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T23:01:29.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:01:29.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:01:29.302 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:01:29.312 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:29.310+0000 7f6ef1f10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:29.314 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:29.312+0000 7f6ef1f10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:29.314 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:29.312+0000 7f6ef1f10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:29.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T23:01:29.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:29.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T23:01:29.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:29.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:29.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:29.393 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:29.394 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:29.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:29.394 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:29.487 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:30.418 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:30.416+0000 7f6ef1f10780 -1 Falling back to public interface 2026-02-22T23:01:30.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:30.488 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:30.489 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:30.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:01:30.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:30.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:30.585 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:31.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:31.586 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:31.587 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:31.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:01:31.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:31.587 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:31.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:32.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:32.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:32.687 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:01:32.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:01:32.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:32.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:32.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:33.374 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:33.372+0000 7f6ef1f10780 -1 osd.0 22 log_to_monitors true 2026-02-22T23:01:33.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:33.783 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:33.784 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:01:33.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:01:33.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:33.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:33.882 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/3765471495,v1:127.0.0.1:6801/3765471495] [v2:127.0.0.1:6802/3765471495,v1:127.0.0.1:6803/3765471495] exists,up 726f6803-5366-42e8-bd23-425e07cdf22a 2026-02-22T23:01:33.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:01:33.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:01:33.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:01:33.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-22T23:01:33.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-22T23:01:33.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-22T23:01:33.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-22T23:01:33.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-22T23:01:33.981 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:34.076 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2026-02-22T23:01:34.077 INFO:tasks.workunit.client.0.trial123.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2026-02-22T23:01:34.077 INFO:tasks.workunit.client.0.trial123.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2026-02-22T23:01:34.077 INFO:tasks.workunit.client.0.trial123.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2026-02-22T23:01:34.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:34.083 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:01:34.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:34.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2026-02-22T23:01:34.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:34.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2026-02-22T23:01:34.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2026-02-22T23:01:34.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2026-02-22T23:01:34.480 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:01:34.481 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:01:34.589 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:01:34.589 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:01:34.590 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:01:34.590 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:01:34.591 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:01:34.592 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:01:34.592 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:01:34.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:01:34.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:01:34.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:01:34.594 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:01:34.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:01:34.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:01:34.595 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:01:34.604 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:01:34.604 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.604 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.604 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:01:34.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:01:34.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:01:34.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2026-02-22T23:01:34.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2026-02-22T23:01:34.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2026-02-22T23:01:34.606 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:01:34.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:01:34.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:01:34.608 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:01:34.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:01:34.609 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:01:34.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:01:34.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:01:34.610 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:01:34.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:01:34.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:01:34.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:01:34.611 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:01:34.611 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:01:34.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:01:34.612 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:01:34.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:01:34.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.613 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.613 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:01:34.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:01:34.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:01:34.614 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2026-02-22T23:01:34.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T23:01:34.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.615 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.615 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189625 2026-02-22T23:01:34.616 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:01:34.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:01:34.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:01:34.619 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:01:34.619 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:01:34.620 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:01:34.620 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:01:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:01:34.621 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:01:34.621 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:01:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:01:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:01:34.622 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:01:34.622 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:01:34.623 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:01:34.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:01:34.624 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:01:34.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:01:34.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.625 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.625 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:01:34.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:01:34.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:01:34.626 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2026-02-22T23:01:34.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T23:01:34.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.627 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.627 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189625 2026-02-22T23:01:34.628 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2026-02-22T23:01:34.629 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-22T23:01:34.649 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T23:01:34.649 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:34.649 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:34.650 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:34.650 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.650 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.651 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:34.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-22T23:01:34.677 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T23:01:34.678 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T23:01:34.678 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:01:34.678 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T23:01:34.678 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T23:01:34.678 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T23:01:34.679 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T23:01:34.679 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T23:01:34.679 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T23:01:34.680 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:01:34.680 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.680 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.680 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.a.asok 2026-02-22T23:01:34.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:01:34.680 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.a.asok config get fsid 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T23:01:34.726 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:01:34.727 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.727 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.727 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.a.asok 2026-02-22T23:01:34.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:01:34.727 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.a.asok config get mon_host 2026-02-22T23:01:34.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-22T23:01:34.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2026-02-22T23:01:34.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T23:01:34.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T23:01:34.771 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T23:01:34.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2026-02-22T23:01:34.772 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-22T23:01:34.790 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T23:01:34.790 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:34.790 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:34.790 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:34.790 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.790 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.791 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:34.791 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-22T23:01:34.817 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:01:34.819 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.820 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.820 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:01:34.820 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:01:34.820 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get fsid 2026-02-22T23:01:34.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T23:01:34.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:01:34.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:01:34.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T23:01:34.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:01:34.866 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get mon_host 2026-02-22T23:01:34.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2026-02-22T23:01:34.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:01:34.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:01:34.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T23:01:34.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:01:34.914 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:01:34.915 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:01:34.916 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b62ff755-b337-434f-9dbd-0307f7c3f05a 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b62ff755-b337-434f-9dbd-0307f7c3f05a' 2026-02-22T23:01:34.917 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 b62ff755-b337-434f-9dbd-0307f7c3f05a 2026-02-22T23:01:34.918 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:01:34.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOiptpPjUENxAAPJEUiKwt6wMI8JX428nGig== 2026-02-22T23:01:34.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOiptpPjUENxAAPJEUiKwt6wMI8JX428nGig=="}' 2026-02-22T23:01:34.925 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b62ff755-b337-434f-9dbd-0307f7c3f05a -i td/ver-health/0/new.json 2026-02-22T23:01:41.029 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:41.035 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2026-02-22T23:01:41.036 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOiptpPjUENxAAPJEUiKwt6wMI8JX428nGig== --osd-uuid b62ff755-b337-434f-9dbd-0307f7c3f05a 2026-02-22T23:01:41.053 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:41.051+0000 7fe9f2c20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:41.063 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:41.053+0000 7fe9f2c20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:41.063 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:41.054+0000 7fe9f2c20780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:41.063 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:41.054+0000 7fe9f2c20780 -1 bdev(0x5593068e9c00 td/ver-health/0/block) open stat got: (1) Operation not permitted 2026-02-22T23:01:41.063 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:41.054+0000 7fe9f2c20780 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2026-02-22T23:01:43.414 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2026-02-22T23:01:43.414 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:01:43.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T23:01:43.415 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T23:01:43.415 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:01:43.518 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T23:01:43.519 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T23:01:43.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:43.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:01:43.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:01:43.520 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:01:43.534 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:43.531+0000 7fbec0433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:43.535 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:43.532+0000 7fbec0433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:43.536 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:43.533+0000 7fbec0433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:43.616 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T23:01:43.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:43.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T23:01:43.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:43.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:43.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:43.617 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:43.617 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:43.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:43.618 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:43.710 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:44.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:44.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:44.711 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:44.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:01:44.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:44.711 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:44.806 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:45.116 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:45.113+0000 7fbec0433780 -1 Falling back to public interface 2026-02-22T23:01:45.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:45.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:45.807 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:01:45.807 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:45.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:45.808 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:45.904 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:46.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:46.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:46.906 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:01:46.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:01:46.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:46.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:47.002 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:47.812 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:47.809+0000 7fbec0433780 -1 osd.0 0 log_to_monitors true 2026-02-22T23:01:48.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:48.004 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:48.004 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:01:48.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:01:48.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:48.005 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:48.125 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:49.126 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:49.127 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:49.127 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T23:01:49.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T23:01:49.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:49.128 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:01:49.224 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3618420493,v1:127.0.0.1:6801/3618420493] [v2:127.0.0.1:6802/3618420493,v1:127.0.0.1:6803/3618420493] exists,up b62ff755-b337-434f-9dbd-0307f7c3f05a 2026-02-22T23:01:49.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:01:49.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:01:49.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:01:49.228 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-22T23:01:49.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:01:49.230 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:01:49.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e9f831e-def0-4d60-af4e-052a1826b2bc 2026-02-22T23:01:49.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0e9f831e-def0-4d60-af4e-052a1826b2bc' 2026-02-22T23:01:49.232 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 0e9f831e-def0-4d60-af4e-052a1826b2bc 2026-02-22T23:01:49.232 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:01:49.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdiptpUzU6DhAA5iK/w+XsKTBTH0oG7my4lQ== 2026-02-22T23:01:49.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdiptpUzU6DhAA5iK/w+XsKTBTH0oG7my4lQ=="}' 2026-02-22T23:01:49.241 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e9f831e-def0-4d60-af4e-052a1826b2bc -i td/ver-health/1/new.json 2026-02-22T23:01:49.343 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:49.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2026-02-22T23:01:49.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdiptpUzU6DhAA5iK/w+XsKTBTH0oG7my4lQ== --osd-uuid 0e9f831e-def0-4d60-af4e-052a1826b2bc 2026-02-22T23:01:49.367 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:49.364+0000 7f8986f3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:49.368 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:49.366+0000 7f8986f3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:49.369 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:49.367+0000 7f8986f3f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:49.370 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:49.367+0000 7f8986f3f780 -1 bdev(0x564a5fc6bc00 td/ver-health/1/block) open stat got: (1) Operation not permitted 2026-02-22T23:01:49.370 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:49.367+0000 7f8986f3f780 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2026-02-22T23:01:51.489 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2026-02-22T23:01:51.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:01:51.490 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T23:01:51.491 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T23:01:51.491 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:01:51.593 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T23:01:51.593 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T23:01:51.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:51.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:01:51.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:01:51.594 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:01:51.609 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:51.607+0000 7f6214a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:51.611 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:51.608+0000 7f6214a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:51.612 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:51.609+0000 7f6214a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:51.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T23:01:51.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:51.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T23:01:51.687 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:51.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:51.688 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:51.688 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:51.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:51.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:51.689 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:51.782 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:52.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:52.784 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:52.784 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:01:52.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:01:52.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:52.785 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:52.878 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:52.952 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:52.950+0000 7f6214a0d780 -1 Falling back to public interface 2026-02-22T23:01:53.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:53.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:53.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:01:53.879 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:53.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:53.879 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:53.973 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:54.975 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:54.975 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:54.975 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:01:54.976 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:01:54.976 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:54.976 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:55.070 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:55.648 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:55.646+0000 7f6214a0d780 -1 osd.1 0 log_to_monitors true 2026-02-22T23:01:56.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:56.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:56.071 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:01:56.072 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:01:56.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:56.072 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:56.172 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:01:57.173 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:01:57.174 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:57.174 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T23:01:57.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T23:01:57.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:57.175 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:01:57.270 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3050005008,v1:127.0.0.1:6809/3050005008] [v2:127.0.0.1:6810/3050005008,v1:127.0.0.1:6811/3050005008] exists,up 0e9f831e-def0-4d60-af4e-052a1826b2bc 2026-02-22T23:01:57.274 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:01:57.274 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:01:57.274 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:01:57.274 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-22T23:01:57.275 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:01:57.276 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:01:57.277 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3a30d5c1-f767-430d-843d-4175b9c573c9 2026-02-22T23:01:57.278 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 3a30d5c1-f767-430d-843d-4175b9c573c9 2026-02-22T23:01:57.278 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3a30d5c1-f767-430d-843d-4175b9c573c9' 2026-02-22T23:01:57.278 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:01:57.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDliptpX0bkEBAADSS7Sulcc4a6yTEaAwTLRQ== 2026-02-22T23:01:57.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDliptpX0bkEBAADSS7Sulcc4a6yTEaAwTLRQ=="}' 2026-02-22T23:01:57.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3a30d5c1-f767-430d-843d-4175b9c573c9 -i td/ver-health/2/new.json 2026-02-22T23:01:57.386 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:01:57.391 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2026-02-22T23:01:57.392 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDliptpX0bkEBAADSS7Sulcc4a6yTEaAwTLRQ== --osd-uuid 3a30d5c1-f767-430d-843d-4175b9c573c9 2026-02-22T23:01:57.410 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:57.408+0000 7f0fae807780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:57.411 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:57.409+0000 7f0fae807780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:57.412 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:57.410+0000 7f0fae807780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:57.412 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:57.410+0000 7f0fae807780 -1 bdev(0x55b5fe7abc00 td/ver-health/2/block) open stat got: (1) Operation not permitted 2026-02-22T23:01:57.412 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:57.410+0000 7f0fae807780 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2026-02-22T23:01:59.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2026-02-22T23:01:59.780 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:01:59.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T23:01:59.781 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T23:01:59.781 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:01:59.882 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T23:01:59.882 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T23:01:59.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:01:59.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:01:59.883 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:01:59.884 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:01:59.899 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:59.897+0000 7f1ebda1a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:59.900 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:59.898+0000 7f1ebda1a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:59.901 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:01:59.899+0000 7f1ebda1a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:01:59.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T23:01:59.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:01:59.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T23:01:59.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:01:59.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:01:59.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:01:59.978 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:01:59.979 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:01:59.979 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:01:59.979 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:00.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:01.073 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:01.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:01.074 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:02:01.074 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:02:01.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:01.075 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:01.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:01.484 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:01.482+0000 7f1ebda1a780 -1 Falling back to public interface 2026-02-22T23:02:02.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:02.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:02.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:02:02.170 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:02:02.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:02.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:02.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:03.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:03.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:03.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:02:03.268 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:02:03.268 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:03.269 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:03.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:04.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:04.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:04.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:02:04.365 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:02:04.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:04.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:04.449 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:04.447+0000 7f1ebda1a780 -1 osd.2 0 log_to_monitors true 2026-02-22T23:02:04.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:05.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:05.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:05.463 INFO:tasks.workunit.client.0.trial123.stdout:5 2026-02-22T23:02:05.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T23:02:05.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:05.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:05.563 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/3194393590,v1:127.0.0.1:6817/3194393590] [v2:127.0.0.1:6818/3194393590,v1:127.0.0.1:6819/3194393590] exists,up 3a30d5c1-f767-430d-843d-4175b9c573c9 2026-02-22T23:02:05.564 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:02:05.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:02:05.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:02:05.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2026-02-22T23:02:05.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2026-02-22T23:02:05.565 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T23:02:05.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T23:02:05.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T23:02:05.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2026-02-22T23:02:05.566 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T23:02:05.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T23:02:05.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:05.667 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:05.668 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:05.668 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:05.668 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:05.668 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:05.668 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T23:02:05.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T23:02:05.685 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2026-02-22T23:02:05.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2026-02-22T23:02:05.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T23:02:05.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2026-02-22T23:02:05.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T23:02:05.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2026-02-22T23:02:05.686 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T23:02:05.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T23:02:05.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:05.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:05.785 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:05.785 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:05.785 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:05.785 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:05.786 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T23:02:05.786 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T23:02:05.804 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:05.805 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:05.822 INFO:tasks.workunit.client.0.trial123.stdout:starting mds.m at 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:05.825 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:05.841 INFO:tasks.workunit.client.0.trial123.stdout:starting mds.n at 2026-02-22T23:02:05.844 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2026-02-22T23:02:10.845 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:11.089 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_OK 2026-02-22T23:02:11.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:11.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2026-02-22T23:02:11.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2026-02-22T23:02:11.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:11.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:11.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:11.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:11.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:11.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:11.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-22T23:02:11.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2026-02-22T23:02:11.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2026-02-22T23:02:11.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T23:02:11.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T23:02:11.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T23:02:11.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2026-02-22T23:02:11.460 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2026-02-22T23:02:11.472 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:11.470+0000 7f99003dad80 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2026-02-22T23:02:11.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T23:02:11.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:11.474 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:11.474 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:11.475 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:11.475 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:11.475 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:11.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2026-02-22T23:02:11.506 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:02:11.508 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:02:11.509 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:02:11.509 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:11.509 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:11.509 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:02:11.509 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:02:11.509 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get fsid 2026-02-22T23:02:11.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T23:02:11.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:02:11.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:02:11.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T23:02:11.551 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:02:11.552 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get mon_host 2026-02-22T23:02:11.599 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2026-02-22T23:02:11.599 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:11.600 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:11.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:11.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:11.600 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:11.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:11.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-22T23:02:11.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2026-02-22T23:02:11.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2026-02-22T23:02:11.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2026-02-22T23:02:11.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2026-02-22T23:02:11.707 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:11.708 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:11.709 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:11.724 INFO:tasks.workunit.client.0.trial123.stdout:starting mds.m at 2026-02-22T23:02:11.727 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:11.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:11.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:12.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:12.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:12.956 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:12.956 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:13.181 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:14.182 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:14.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:14.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:14.183 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:14.408 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:15.409 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:15.410 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:15.638 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:16.639 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:16.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:16.640 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:16.641 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:16.868 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:17.869 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:17.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:17.870 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:17.871 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:18.097 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:19.098 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:19.098 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:19.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:19.099 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:19.323 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:20.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:20.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:20.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:20.325 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:20.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-22T23:02:20.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-22T23:02:20.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-22T23:02:20.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-22T23:02:20.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-22T23:02:20.549 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:20.795 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_WARN There are daemons running an older version of ceph 2026-02-22T23:02:20.795 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2026-02-22T23:02:20.795 INFO:tasks.workunit.client.0.trial123.stdout: mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test 2026-02-22T23:02:20.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:20.802 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-22T23:02:21.054 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:21.055 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2026-02-22T23:02:21.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:21.306 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2026-02-22T23:02:21.557 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:153: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.2 2026-02-22T23:02:21.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:21.558 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:21.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:21.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:21.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:21.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: activate_osd td/ver-health 2 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:21.665 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:02:21.666 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2026-02-22T23:02:21.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T23:02:21.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:02:21.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:02:21.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:02:21.682 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:21.680+0000 7f9119553780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:21.684 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:21.681+0000 7f9119553780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:21.685 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:21.682+0000 7f9119553780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:21.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T23:02:21.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:02:21.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T23:02:21.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:02:21.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:02:21.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:21.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:02:21.857 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:02:21.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:21.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:22.043 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:23.026 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:23.023+0000 7f9119553780 -1 Falling back to public interface 2026-02-22T23:02:23.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:23.044 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:23.044 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:02:23.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:02:23.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:23.045 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:23.229 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:24.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:24.231 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:24.231 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:02:24.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:02:24.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:24.232 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:24.419 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:25.420 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:25.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:25.421 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:02:25.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:02:25.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:25.421 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:25.607 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:25.728 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:25.725+0000 7f9119553780 -1 osd.2 15 log_to_monitors true 2026-02-22T23:02:26.608 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:26.609 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:26.609 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:02:26.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:02:26.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:26.610 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 16 last_clean_interval [13,15) [v2:127.0.0.1:6816/1613220659,v1:127.0.0.1:6817/1613220659] [v2:127.0.0.1:6818/1613220659,v1:127.0.0.1:6819/1613220659] exists,up 3a30d5c1-f767-430d-843d-4175b9c573c9 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:155: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.0 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:26.797 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:26.798 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:26.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:26.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: ceph_debug_version_for_testing=02.00.00-gversion-test 2026-02-22T23:02:26.906 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: activate_osd td/ver-health 0 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:02:26.907 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:02:26.908 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:02:26.921 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:26.918+0000 7eff3310e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:26.922 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:26.919+0000 7eff3310e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:26.923 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:26.920+0000 7eff3310e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:27.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T23:02:27.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:02:27.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T23:02:27.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:02:27.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:02:27.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:27.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:02:27.095 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:02:27.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:27.096 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:02:27.281 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:28.257 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:28.255+0000 7eff3310e780 -1 Falling back to public interface 2026-02-22T23:02:28.282 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:28.282 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:28.282 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:02:28.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:02:28.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:28.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:02:28.471 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:29.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:29.473 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:29.474 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:02:29.474 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T23:02:29.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:29.475 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:02:29.666 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:30.667 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:30.668 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:30.668 INFO:tasks.workunit.client.0.trial123.stdout:3 2026-02-22T23:02:30.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T23:02:30.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:30.669 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:02:30.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:30.975 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:30.973+0000 7eff3310e780 -1 osd.0 17 log_to_monitors true 2026-02-22T23:02:31.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:31.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:31.860 INFO:tasks.workunit.client.0.trial123.stdout:4 2026-02-22T23:02:31.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T23:02:31.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:31.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:02:32.046 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 21 up_thru 0 down_at 19 last_clean_interval [5,17) [v2:127.0.0.1:6800/4137068528,v1:127.0.0.1:6801/4137068528] [v2:127.0.0.1:6802/4137068528,v1:127.0.0.1:6803/4137068528] exists,up b62ff755-b337-434f-9dbd-0307f7c3f05a 2026-02-22T23:02:32.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:02:32.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:02:32.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:02:32.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:158: TEST_check_version_health_2: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:02:32.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:02:32.048 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-22T23:02:32.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-22T23:02:32.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-22T23:02:32.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-22T23:02:32.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:32.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:32.049 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:02:32.270 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:33.271 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:33.271 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:33.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:33.272 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:02:33.501 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:02:34.502 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:02:34.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:02:34.503 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:02:34.504 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:02:34.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-22T23:02:34.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-22T23:02:34.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-22T23:02:34.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-22T23:02:34.728 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-22T23:02:34.729 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:160: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:34.972 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2026-02-22T23:02:34.973 INFO:tasks.workunit.client.0.trial123.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2026-02-22T23:02:34.973 INFO:tasks.workunit.client.0.trial123.stdout: mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test 2026-02-22T23:02:34.973 INFO:tasks.workunit.client.0.trial123.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2026-02-22T23:02:34.980 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: ceph health 2026-02-22T23:02:34.980 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:02:35.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:35.206 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:02:35.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:35.455 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2026-02-22T23:02:35.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:35.713 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: grep -q 'mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test' 2026-02-22T23:02:35.969 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: ceph health detail 2026-02-22T23:02:35.970 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2026-02-22T23:02:36.223 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:36.224 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:36.332 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:36.333 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:02:36.333 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:02:36.334 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:02:36.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:02:36.335 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:02:36.336 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:02:36.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:02:36.336 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:02:36.337 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:02:36.337 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:02:36.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:02:36.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:02:36.338 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:02:36.349 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:02:36.349 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.349 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.349 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:02:36.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:02:36.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:02:36.351 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:36.352 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:36.353 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:36.353 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:02:36.354 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:02:36.354 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:02:36.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:02:36.355 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:02:36.355 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:02:36.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:02:36.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:02:36.356 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:02:36.357 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:02:36.357 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:02:36.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:02:36.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:02:36.359 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:02:36.359 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.359 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.359 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:02:36.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:02:36.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:02:36.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2026-02-22T23:02:36.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T23:02:36.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.361 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189625 2026-02-22T23:02:36.364 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T23:02:36.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T23:02:36.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T23:02:36.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-22T23:02:36.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_3 td/ver-health 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:170: TEST_check_version_health_3: local dir=td/ver-health 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:173: TEST_check_version_health_3: CEPH_ARGS='--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:175: TEST_check_version_health_3: setup td/ver-health 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:36.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:36.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:36.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:36.367 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:02:36.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:02:36.367 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:02:36.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:02:36.368 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:02:36.368 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:02:36.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:02:36.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:02:36.369 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:02:36.369 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:02:36.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:02:36.370 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:02:36.371 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:02:36.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:02:36.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.372 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.372 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:02:36.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:02:36.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:02:36.373 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2026-02-22T23:02:36.374 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T23:02:36.374 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.374 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189625 2026-02-22T23:02:36.375 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T23:02:36.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T23:02:36.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T23:02:36.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-22T23:02:36.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:178: TEST_check_version_health_3: run_mon td/ver-health a --public-addr=127.0.0.1:7165 2026-02-22T23:02:36.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2026-02-22T23:02:36.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T23:02:36.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T23:02:36.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T23:02:36.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2026-02-22T23:02:36.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 2026-02-22T23:02:36.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T23:02:36.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:36.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:36.396 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:36.396 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.396 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.396 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:36.397 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 2026-02-22T23:02:36.422 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T23:02:36.423 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T23:02:36.423 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:02:36.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T23:02:36.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T23:02:36.424 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T23:02:36.425 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T23:02:36.425 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T23:02:36.425 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T23:02:36.425 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:02:36.425 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.425 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.425 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.a.asok 2026-02-22T23:02:36.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:02:36.426 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.a.asok config get fsid 2026-02-22T23:02:36.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T23:02:36.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:02:36.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T23:02:36.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T23:02:36.471 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T23:02:36.471 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T23:02:36.471 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T23:02:36.472 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T23:02:36.472 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:02:36.472 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.472 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.472 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.a.asok 2026-02-22T23:02:36.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:02:36.472 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.a.asok config get mon_host 2026-02-22T23:02:36.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:179: TEST_check_version_health_3: run_mon td/ver-health b --public-addr=127.0.0.1:7166 2026-02-22T23:02:36.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2026-02-22T23:02:36.514 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T23:02:36.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2026-02-22T23:02:36.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T23:02:36.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2026-02-22T23:02:36.515 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 2026-02-22T23:02:36.532 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T23:02:36.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:36.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:36.533 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:36.533 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.533 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.533 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:36.533 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 2026-02-22T23:02:36.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T23:02:36.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2026-02-22T23:02:36.562 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:02:36.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:02:36.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T23:02:36.563 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T23:02:36.563 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:02:36.563 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:02:36.563 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:02:36.564 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:02:36.564 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.564 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.564 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:02:36.564 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:02:36.564 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get fsid 2026-02-22T23:02:36.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2026-02-22T23:02:36.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T23:02:36.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2026-02-22T23:02:36.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T23:02:36.605 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T23:02:36.606 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2026-02-22T23:02:36.606 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2026-02-22T23:02:36.606 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2026-02-22T23:02:36.606 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T23:02:36.607 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.607 INFO:tasks.workunit.client.0.trial123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.607 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189625/ceph-mon.b.asok 2026-02-22T23:02:36.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T23:02:36.607 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189625/ceph-mon.b.asok config get mon_host 2026-02-22T23:02:36.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:181: TEST_check_version_health_3: local start_osd_time=113 2026-02-22T23:02:36.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-22T23:02:36.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: run_osd td/ver-health 0 2026-02-22T23:02:36.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:02:36.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:02:36.650 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:36.651 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2026-02-22T23:02:36.652 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:02:36.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72dffa10-7abb-46a1-83e9-5e3bff068022 2026-02-22T23:02:36.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 72dffa10-7abb-46a1-83e9-5e3bff068022' 2026-02-22T23:02:36.653 INFO:tasks.workunit.client.0.trial123.stdout:add osd0 72dffa10-7abb-46a1-83e9-5e3bff068022 2026-02-22T23:02:36.653 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:02:36.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMi5tpm0eCJxAAjrVhZEE2zCYW5HKS/1r5hQ== 2026-02-22T23:02:36.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMi5tpm0eCJxAAjrVhZEE2zCYW5HKS/1r5hQ=="}' 2026-02-22T23:02:36.664 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72dffa10-7abb-46a1-83e9-5e3bff068022 -i td/ver-health/0/new.json 2026-02-22T23:02:42.767 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:02:42.773 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2026-02-22T23:02:42.774 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMi5tpm0eCJxAAjrVhZEE2zCYW5HKS/1r5hQ== --osd-uuid 72dffa10-7abb-46a1-83e9-5e3bff068022 2026-02-22T23:02:42.792 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.790+0000 7f380a418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:42.793 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.791+0000 7f380a418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:42.794 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.792+0000 7f380a418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:42.795 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.793+0000 7f380a418780 -1 memstore(td/ver-health/0) td/ver-health/0 2026-02-22T23:02:42.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2026-02-22T23:02:42.800 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:02:42.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T23:02:42.801 INFO:tasks.workunit.client.0.trial123.stdout:adding osd0 key to auth repository 2026-02-22T23:02:42.801 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:02:42.900 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T23:02:42.900 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T23:02:42.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:02:42.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:02:42.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:02:42.901 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:02:42.916 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.914+0000 7f2a65c31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:42.918 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.916+0000 7f2a65c31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:42.919 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.917+0000 7f2a65c31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:42.920 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.918+0000 7f2a65c31780 -1 Falling back to public interface 2026-02-22T23:02:42.926 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:42.924+0000 7f2a65c31780 -1 osd.0 0 log_to_monitors true 2026-02-22T23:02:42.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T23:02:42.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:02:42.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T23:02:42.998 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:02:42.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:02:42.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:42.999 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:02:42.999 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:02:43.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:43.000 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:02:43.093 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:44.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:44.094 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:44.095 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:02:44.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:02:44.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:44.095 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:02:44.192 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1058859380,v1:127.0.0.1:6801/1058859380] [v2:127.0.0.1:6802/1058859380,v1:127.0.0.1:6803/1058859380] exists,up 72dffa10-7abb-46a1-83e9-5e3bff068022 2026-02-22T23:02:44.195 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:02:44.195 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: run_osd td/ver-health 1 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:02:44.196 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:02:44.197 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97bf422e-a09c-42de-bec0-bb58dd59e0c4 2026-02-22T23:02:44.198 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 97bf422e-a09c-42de-bec0-bb58dd59e0c4' 2026-02-22T23:02:44.199 INFO:tasks.workunit.client.0.trial123.stdout:add osd1 97bf422e-a09c-42de-bec0-bb58dd59e0c4 2026-02-22T23:02:44.199 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:02:44.207 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUi5tp+i0+DBAA91KYoNF8nZ38oeSedxVsNA== 2026-02-22T23:02:44.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUi5tp+i0+DBAA91KYoNF8nZ38oeSedxVsNA=="}' 2026-02-22T23:02:44.208 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97bf422e-a09c-42de-bec0-bb58dd59e0c4 -i td/ver-health/1/new.json 2026-02-22T23:02:44.311 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:02:44.317 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2026-02-22T23:02:44.318 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUi5tp+i0+DBAA91KYoNF8nZ38oeSedxVsNA== --osd-uuid 97bf422e-a09c-42de-bec0-bb58dd59e0c4 2026-02-22T23:02:44.336 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.334+0000 7fbe98033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:44.337 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.336+0000 7fbe98033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:44.338 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.337+0000 7fbe98033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:44.339 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.337+0000 7fbe98033780 -1 memstore(td/ver-health/1) td/ver-health/1 2026-02-22T23:02:44.344 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2026-02-22T23:02:44.344 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:02:44.345 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T23:02:44.345 INFO:tasks.workunit.client.0.trial123.stdout:adding osd1 key to auth repository 2026-02-22T23:02:44.345 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:02:44.448 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T23:02:44.449 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T23:02:44.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:02:44.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:02:44.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:02:44.450 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:02:44.465 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.463+0000 7ff90d418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:44.466 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.464+0000 7ff90d418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:44.467 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.465+0000 7ff90d418780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:44.469 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.467+0000 7ff90d418780 -1 Falling back to public interface 2026-02-22T23:02:44.476 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:44.474+0000 7ff90d418780 -1 osd.1 0 log_to_monitors true 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:02:44.546 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:02:44.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:44.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:02:44.642 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:45.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:45.643 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:45.644 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:02:45.644 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:02:45.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:45.645 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:02:45.740 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2232032096,v1:127.0.0.1:6809/2232032096] [v2:127.0.0.1:6810/2232032096,v1:127.0.0.1:6811/2232032096] exists,up 97bf422e-a09c-42de-bec0-bb58dd59e0c4 2026-02-22T23:02:45.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:02:45.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:02:45.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:02:45.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-22T23:02:45.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: run_osd td/ver-health 2 2026-02-22T23:02:45.745 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-22T23:02:45.746 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:02:45.747 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca56db07-980b-4296-b33e-cfaec10159a1 2026-02-22T23:02:45.748 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ca56db07-980b-4296-b33e-cfaec10159a1' 2026-02-22T23:02:45.749 INFO:tasks.workunit.client.0.trial123.stdout:add osd2 ca56db07-980b-4296-b33e-cfaec10159a1 2026-02-22T23:02:45.749 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T23:02:45.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVi5tpYP3vLBAAwtUB68zIEzkuXFO7Tgedag== 2026-02-22T23:02:45.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVi5tpYP3vLBAAwtUB68zIEzkuXFO7Tgedag=="}' 2026-02-22T23:02:45.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca56db07-980b-4296-b33e-cfaec10159a1 -i td/ver-health/2/new.json 2026-02-22T23:02:45.856 INFO:tasks.workunit.client.0.trial123.stdout:2 2026-02-22T23:02:45.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2026-02-22T23:02:45.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVi5tpYP3vLBAAwtUB68zIEzkuXFO7Tgedag== --osd-uuid ca56db07-980b-4296-b33e-cfaec10159a1 2026-02-22T23:02:45.880 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:45.878+0000 7f3906fd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:45.882 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:45.880+0000 7f3906fd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:45.883 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:45.881+0000 7f3906fd4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:45.883 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:45.881+0000 7f3906fd4780 -1 memstore(td/ver-health/2) td/ver-health/2 2026-02-22T23:02:45.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2026-02-22T23:02:45.888 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T23:02:45.889 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T23:02:45.889 INFO:tasks.workunit.client.0.trial123.stdout:adding osd2 key to auth repository 2026-02-22T23:02:45.890 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T23:02:45.992 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T23:02:45.993 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T23:02:45.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:02:45.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T23:02:45.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T23:02:45.993 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T23:02:46.009 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:46.007+0000 7f0f91e82780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:46.010 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:46.008+0000 7f0f91e82780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:46.011 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:46.009+0000 7f0f91e82780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:46.012 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:46.010+0000 7f0f91e82780 -1 Falling back to public interface 2026-02-22T23:02:46.019 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:46.017+0000 7f0f91e82780 -1 osd.2 0 log_to_monitors true 2026-02-22T23:02:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T23:02:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:02:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T23:02:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:02:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:02:46.091 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:46.092 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:02:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:02:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:46.092 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:46.185 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T23:02:47.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T23:02:47.186 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:47.186 INFO:tasks.workunit.client.0.trial123.stdout:1 2026-02-22T23:02:47.187 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T23:02:47.187 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:47.187 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:02:47.283 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/4164362270,v1:127.0.0.1:6817/4164362270] [v2:127.0.0.1:6818/4164362270,v1:127.0.0.1:6819/4164362270] exists,up ca56db07-980b-4296-b33e-cfaec10159a1 2026-02-22T23:02:47.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:02:47.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:02:47.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:02:47.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:187: TEST_check_version_health_3: local warn_older_version_delay=31 2026-02-22T23:02:47.283 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:189: TEST_check_version_health_3: sleep 5 2026-02-22T23:02:52.285 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:190: TEST_check_version_health_3: ceph health detail 2026-02-22T23:02:52.377 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_OK 2026-02-22T23:02:52.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: ceph health detail 2026-02-22T23:02:52.383 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2026-02-22T23:02:52.480 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:193: TEST_check_version_health_3: ceph tell 'mon.*' injectargs '--mon_warn_older_version_delay 31' 2026-02-22T23:02:52.533 INFO:tasks.workunit.client.0.trial123.stdout:mon.a: {} 2026-02-22T23:02:52.541 INFO:tasks.workunit.client.0.trial123.stdout:mon.b: {} 2026-02-22T23:02:52.547 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:194: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.1 2026-02-22T23:02:52.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:02:52.548 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:02:52.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:02:52.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:02:52.548 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:02:52.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:02:52.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-22T23:02:52.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: activate_osd td/ver-health 1 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:02:52.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:02:52.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2026-02-22T23:02:52.660 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T23:02:52.661 INFO:tasks.workunit.client.0.trial123.stdout:start osd.1 2026-02-22T23:02:52.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:02:52.662 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2026-02-22T23:02:52.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T23:02:52.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:02:52.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:02:52.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:02:52.677 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:52.675+0000 7fc8c5211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:52.678 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:52.676+0000 7fc8c5211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:52.679 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:52.677+0000 7fc8c5211780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:02:52.680 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:52.679+0000 7fc8c5211780 -1 Falling back to public interface 2026-02-22T23:02:52.687 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:02:52.685+0000 7fc8c5211780 -1 osd.1 0 log_to_monitors true 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:02:52.755 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:02:52.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:02:52.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T23:02:52.851 INFO:tasks.workunit.client.0.trial123.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 14 last_clean_interval [0,0) [v2:127.0.0.1:6808/4048257122,v1:127.0.0.1:6809/4048257122] [v2:127.0.0.1:6810/4048257122,v1:127.0.0.1:6811/4048257122] exists,up 97bf422e-a09c-42de-bec0-bb58dd59e0c4 2026-02-22T23:02:52.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:02:52.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:02:52.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:02:52.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:200: TEST_check_version_health_3: sleep 10 2026-02-22T23:03:02.852 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:02.853 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2026-02-22T23:03:02.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:205: TEST_check_version_health_3: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 20 2026-02-22T23:03:02.954 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:02.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-22T23:03:02.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-22T23:03:02.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-22T23:03:02.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-22T23:03:02.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:02.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:02.955 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:03.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:04.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:04.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:04.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:04.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:04.157 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:05.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:05.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:05.158 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:05.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:05.258 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:06.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:06.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:06.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:06.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:06.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:07.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:07.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:07.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:07.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:07.456 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:08.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:08.458 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:08.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:08.459 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:08.558 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:09.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:09.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:09.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:09.560 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:09.657 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:10.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:10.658 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:10.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:10.659 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:10.756 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:11.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:11.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:11.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:11.757 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:11.856 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:12.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:12.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:12.857 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:12.858 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:12.958 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:13.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:13.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:13.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:13.961 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:14.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:15.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:15.058 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:15.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:15.059 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:15.159 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:16.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:16.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:16.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:16.161 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:16.259 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:17.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:17.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:17.260 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:17.261 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:17.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-22T23:03:17.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-22T23:03:17.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-22T23:03:17.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-22T23:03:17.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-22T23:03:17.358 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:207: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:17.451 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2026-02-22T23:03:17.451 INFO:tasks.workunit.client.0.trial123.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2026-02-22T23:03:17.451 INFO:tasks.workunit.client.0.trial123.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2026-02-22T23:03:17.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:17.457 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-22T23:03:17.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:17.554 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2026-02-22T23:03:17.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:17.653 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2026-02-22T23:03:17.750 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:213: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.2 2026-02-22T23:03:17.751 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:03:17.751 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:03:17.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:03:17.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:03:17.751 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:03:17.859 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: activate_osd td/ver-health 2 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2026-02-22T23:03:17.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:03:17.861 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:03:17.862 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:03:17.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:03:17.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:03:17.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:03:17.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2026-02-22T23:03:17.863 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T23:03:17.863 INFO:tasks.workunit.client.0.trial123.stdout:start osd.2 2026-02-22T23:03:17.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:03:17.864 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2026-02-22T23:03:17.864 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T23:03:17.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:03:17.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:03:17.865 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:03:17.877 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:17.875+0000 7fc20ca27780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:03:17.878 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:17.876+0000 7fc20ca27780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:03:17.879 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:17.877+0000 7fc20ca27780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:03:17.880 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:17.879+0000 7fc20ca27780 -1 Falling back to public interface 2026-02-22T23:03:17.889 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:17.887+0000 7fc20ca27780 -1 osd.2 0 log_to_monitors true 2026-02-22T23:03:17.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T23:03:17.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:03:17.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T23:03:17.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:03:17.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:03:17.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:03:17.960 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:03:17.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:03:17.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:03:17.960 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T23:03:18.055 INFO:tasks.workunit.client.0.trial123.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 18 last_clean_interval [0,0) [v2:127.0.0.1:6816/3552938698,v1:127.0.0.1:6817/3552938698] [v2:127.0.0.1:6818/3552938698,v1:127.0.0.1:6819/3552938698] exists,up ca56db07-980b-4296-b33e-cfaec10159a1 2026-02-22T23:03:18.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:215: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.0 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:03:18.057 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:03:18.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:03:18.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: ceph_debug_version_for_testing=02.00.00-gversion-test 2026-02-22T23:03:18.165 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: activate_osd td/ver-health 0 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T23:03:18.166 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T23:03:18.167 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T23:03:18.168 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T23:03:18.169 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2026-02-22T23:03:18.169 INFO:tasks.workunit.client.0.trial123.stdout:start osd.0 2026-02-22T23:03:18.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2026-02-22T23:03:18.170 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=10aeb903-ded8-40f7-bbbc-3f55990210ab --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.189625/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T23:03:18.171 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2026-02-22T23:03:18.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2026-02-22T23:03:18.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T23:03:18.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T23:03:18.171 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T23:03:18.181 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:18.179+0000 7fe4a8ff9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:03:18.183 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:18.181+0000 7fe4a8ff9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:03:18.183 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:18.182+0000 7fe4a8ff9780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T23:03:18.185 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:18.183+0000 7fe4a8ff9780 -1 Falling back to public interface 2026-02-22T23:03:18.191 INFO:tasks.workunit.client.0.trial123.stderr:2026-02-22T23:03:18.189+0000 7fe4a8ff9780 -1 osd.0 0 log_to_monitors true 2026-02-22T23:03:18.266 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stdout:0 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T23:03:18.267 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T23:03:18.362 INFO:tasks.workunit.client.0.trial123.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [0,0) [v2:127.0.0.1:6800/2788920375,v1:127.0.0.1:6801/2788920375] [v2:127.0.0.1:6802/2788920375,v1:127.0.0.1:6803/2788920375] exists,up 72dffa10-7abb-46a1-83e9-5e3bff068022 2026-02-22T23:03:18.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T23:03:18.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T23:03:18.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T23:03:18.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:218: TEST_check_version_health_3: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:03:18.364 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:03:18.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-22T23:03:18.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-22T23:03:18.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-22T23:03:18.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-22T23:03:18.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:18.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:18.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:03:18.461 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:19.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:19.463 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:19.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:19.464 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:03:19.561 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:20.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:20.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:20.562 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:20.563 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:03:20.661 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-22T23:03:21.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-22T23:03:21.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-22T23:03:21.662 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-22T23:03:21.663 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:03:21.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-22T23:03:21.762 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-22T23:03:21.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-22T23:03:21.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-22T23:03:21.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-22T23:03:21.763 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:220: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:21.854 INFO:tasks.workunit.client.0.trial123.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2026-02-22T23:03:21.854 INFO:tasks.workunit.client.0.trial123.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2026-02-22T23:03:21.854 INFO:tasks.workunit.client.0.trial123.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2026-02-22T23:03:21.855 INFO:tasks.workunit.client.0.trial123.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2026-02-22T23:03:21.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:21.860 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-22T23:03:21.958 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:21.959 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2026-02-22T23:03:22.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:22.056 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2026-02-22T23:03:22.153 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: ceph health detail 2026-02-22T23:03:22.154 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:03:22.252 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:03:22.360 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:03:22.361 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:03:22.362 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:03:22.362 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:03:22.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:03:22.363 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:03:22.363 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:03:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:03:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:03:22.365 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:03:22.365 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:03:22.366 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:03:22.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T23:03:22.367 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:03:22.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:03:22.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:03:22.373 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:03:22.374 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:03:22.375 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/ver-health 0 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T23:03:22.376 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T23:03:22.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T23:03:22.377 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T23:03:22.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T23:03:22.378 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T23:03:22.378 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T23:03:22.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T23:03:22.379 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T23:03:22.379 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T23:03:22.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:03:22.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T23:03:22.380 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T23:03:22.380 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T23:03:22.381 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:03:22.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T23:03:22.382 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2026-02-22T23:03:22.383 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T23:03:22.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T23:03:22.384 INFO:tasks.workunit.client.0.trial123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189625 2026-02-22T23:03:22.384 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189625 2026-02-22T23:03:22.385 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T23:03:22.385 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T23:03:22.386 INFO:tasks.workunit.client.0.trial123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T23:03:22.386 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T23:03:22.386 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T23:03:22.412 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2026-02-22T23:03:22.412 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-22T23:03:22.758 DEBUG:teuthology.parallel:result is None 2026-02-22T23:03:22.758 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T23:03:22.782 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T23:03:22.782 DEBUG:teuthology.orchestra.run.trial123:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-22T23:03:22.851 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T23:03:22.851 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T23:03:22.881 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T23:03:22.882 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T23:03:22.991 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T23:03:23.002 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T23:03:23.002 DEBUG:teuthology.orchestra.run.trial123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T23:03:23.018 INFO:teuthology.orchestra.run.trial123.stderr:bash: line 1: ntpq: command not found 2026-02-22T23:03:23.051 INFO:teuthology.orchestra.run.trial123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-22T23:03:23.051 INFO:teuthology.orchestra.run.trial123.stdout:=============================================================================== 2026-02-22T23:03:23.051 INFO:teuthology.orchestra.run.trial123.stdout:^- LAX.CALTICK.NET 2 8 377 80 -1211us[-1224us] +/- 33ms 2026-02-22T23:03:23.051 INFO:teuthology.orchestra.run.trial123.stdout:^* 50.205.57.38 1 7 377 75 +188us[ +174us] +/- 5223us 2026-02-22T23:03:23.051 INFO:teuthology.orchestra.run.trial123.stdout:^- clock02.cymru.com 2 6 377 8 -7345us[-7345us] +/- 112ms 2026-02-22T23:03:23.051 INFO:teuthology.orchestra.run.trial123.stdout:^- c-73-238-61-241.hsd1.nh.> 2 8 377 11 +1832us[+1832us] +/- 61ms 2026-02-22T23:03:23.051 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T23:03:23.062 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T23:03:23.062 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T23:03:23.076 DEBUG:teuthology.orchestra.run.trial123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-22T23:03:23.145 DEBUG:teuthology.orchestra.run.trial123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2026-02-22T23:03:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:03:23.170 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T23:03:23.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T23:03:23.198 INFO:teuthology.task.internal:Duration was 2633.142686 seconds 2026-02-22T23:03:23.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T23:03:23.213 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T23:03:23.213 DEBUG:teuthology.orchestra.run.trial123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T23:03:23.251 INFO:teuthology.orchestra.run.trial123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-22T23:03:23.538 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T23:03:23.538 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial123.front.sepia.ceph.com 2026-02-22T23:03:23.539 DEBUG:teuthology.orchestra.run.trial123:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T23:03:23.597 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T23:03:23.597 DEBUG:teuthology.orchestra.run.trial123:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T23:03:23.696 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T23:03:23.701 DEBUG:teuthology.orchestra.run.trial123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T23:03:23.728 INFO:teuthology.orchestra.run.trial123.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T23:03:23.729 INFO:teuthology.orchestra.run.trial123.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T23:03:23.729 INFO:teuthology.orchestra.run.trial123.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T23:03:23.729 INFO:teuthology.orchestra.run.trial123.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T23:03:23.729 INFO:teuthology.orchestra.run.trial123.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T23:03:23.739 INFO:teuthology.orchestra.run.trial123.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 81.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T23:03:23.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T23:03:23.778 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T23:03:23.779 DEBUG:teuthology.orchestra.run.trial123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T23:03:23.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T23:03:23.831 DEBUG:teuthology.orchestra.run.trial123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:03:23.880 INFO:teuthology.orchestra.run.trial123.stdout:kernel.core_pattern = core 2026-02-22T23:03:23.891 DEBUG:teuthology.orchestra.run.trial123:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T23:03:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T23:03:23.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T23:03:23.963 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T23:03:23.965 DEBUG:teuthology.misc:Transferring archived files from trial123:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64214/remote/trial123 2026-02-22T23:03:23.966 DEBUG:teuthology.orchestra.run.trial123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T23:03:24.023 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T23:03:24.023 DEBUG:teuthology.orchestra.run.trial123:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T23:03:24.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T23:03:24.089 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T23:03:24.089 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T23:03:24.105 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T23:03:24.106 DEBUG:teuthology.orchestra.run.trial123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T23:03:24.131 INFO:teuthology.orchestra.run.trial123.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 23:03 /home/ubuntu/cephtest 2026-02-22T23:03:24.132 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T23:03:24.148 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T23:03:24.182 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 2633.142685651779 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-22T23:03:24.182 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T23:03:24.299 INFO:teuthology.run:pass